Data Use: Controlling non-response bias and item non-response bias using CATI techniques

By Michael Sullivan | November 1, 1991

Reading time: 19 minutes

Abstract

This article describes how computer-assisted telephone interviewing (CATI) techniques can help reduce non-response bias and item non-response bias in survey research. The author presents mixed-mode surveying, which combines telephone and mail surveys, as one approach to controlling and measuring non-response bias.

Research Topics:: Data Analysis | Mail Surveys | Quantitative Research | Telephone Interviewing/CATI
Content Type: Data Use

Share Print

Editor's note: Michael Sullivan is senior partner of Freeman, Sullivan & Company (FSC), a San Francisco-based market research firm specializing in the conduct of surveys. This article is adapted from a presentation made at the 1991 Sawtooth Software Conference.

The title of this article is perhaps a little ambitious. It suggests that computer assisted telephone interviewing (CATI) techniques can be used to control to very troublesome "threats" to the validity of survey measurements. As will be apparent momentarily, non-response bias and item non-response bias cannot be completely controlled using any currently available techniques. Nevertheless, CATI techniques offer some very powerful and cost effective capabilities for reducing these sources of bias in surveys. In addition to facilitating tight control of non-response bias in telephone surveying, CATI techniques are extremely useful in mixed mode surveying-an approach to controlling survey non-response bias which combines the strengths of two or more survey modes. They also provide technology necessary to efficiently conduct interviews using a measurement technique known as bounded recall, a procedure which greatly reduces item non-response bias. This article will first discuss non-response bias in general. Then, I will present some examples of surveys using mixed mode techniques and bounded recall, focusing on what we think works and what doesn't.

Let's begin by discussing non-response bias in a little more detail.

Non-response bias defined

There is substantial academic literature discussing nonresponse and item non-response bias. A good overview of the problem is presented in "Telephone Survey Methodology" by Robert Groves, et al. (You should read Chapters 12 and 13 of this 1988 book if you want an overview of the subject.) In a nutshell, non-response bias and item non-response bias are exactly what they sound like-bias in survey measurements due either to the fact that a respondent could not be contacted at all, or to the fact that the respondent refused or failed to provide some subset of the information sought by the surveyor.

Non-response is a necessary but not sufficient condition for non-response bias. Non-response bias (item or otherwise) actually has two components. It is made up of the nonresponse rate and the difference between respondents and non-respondents. For simple sample statistics such as means and proportions, non-response bias can be viewed as a simple linear function as follows:

Y_t =Y_r+ (nr/(r+nr) (Y_r Y_nr))

Where: y_t =the "true value" of the sample statistic
y_r=the value of the statistic for the r respondents
y_nr =the value of the statistic for the nr non-respondents

This simple mathematical construct illustrates some interesting properties of non-response bias. First, it is clear that non-response bias is not simply the result of the non-response rate. A survey with a non-response rate of 99 percent may have little or no non-response bias if the difference between the observed and unobserved respondents is little or nothing. The converse is also possible. That is, a survey with a relatively low non-response rate (say 10 to 20 percent) may suffer from significant non-response bias if the difference between observed and unobserved respondents is sufficiently great.

Sources of non-response bias

All major modes of survey contact (in-person, telephone and mail) are susceptible to non-response bias of different degrees and kinds. Until fairly recently the three modes of surveying were presented as competing alternative measurement techniques, with the primary determinant of choice among the alternatives being cost. The conventional wisdom has been that in-person interviewing generally produces superior response rates and data quality, followed by telephone interviewing, followed by mail surveying. However, in recent years, the apparent superiority of in-person interviewing over the other survey modes has been questioned. As systematic studies of non-response bias associated with the different modes accumulate, it has become increasingly clear that the different survey modes experience non-response for different reasons and therefore experience different (and potentially offsetting) non-response biases.

Looking at the possible outcomes of a survey contact, it is apparent that non-response bias can arise in a number of ways. In general, any systematic failure in attempting to survey respondents can result in non-response bias. Such failures can occur for the following important reasons:

1. Initial contact cannot be established with the sampled respondent-because the respondent has moved, is not home, lives in a dangerous neighborhood or is somehow screening contacts with the outside world (for example, using security guards, secretaries or telephone answering machines);

2. Respondents (or their "representatives") refuse to participate in the survey; and

3. Respondents are physically incapacitated or unable to understand and speak any of the languages being used in surveying.

Non-response for some of these reasons is prima facie evidence of the existence of non-response bias.

Respondents who cannot write or speak the languages in which surveying is being conducted are very likely to be systematically different from those who can on a number of dimensions. They are likely to be less wealthy, possess less formal education and be less acculturated than those respondents who write or speak the languages in which the survey is being conducted. If these respondents constitute a significant fraction of the population, failing to include them in the survey will significantly bias survey results.

In some regions of the United States, great care must be taken to control this source of non-response bias. In California, for example, about 8 percent of the general population does not speak English well enough to be interviewed using that language. The fraction of the California population that does not write English well enough to understand and respond to a mail survey is probably substantially higher. To control for non-response bias due to differences in acculturation, interviewing must be routinely conducted in Spanish in statewide surveys; and in some counties it must be conducted in Mandarin, Cantonese or Vietnamese to obtain representative samples. It is more difficult to control for non-response bias due to differences in literacy. Usually, interviewing by telephone or in-person is required in these populations.

Another fairly automatic source of non-response bias is non-response due to respondents living in a dangerous place or to their screening contact with the outside world. In my experience, this sort of bias is most often encountered in urban areas where significant segments of the population live or work in dangerous locations or in high security areas. The problem is particularly acute for in-person survey techniques. In fact, it has been suggested that in many urban areas in the United States, non-response bias due to these factors may favor telephone interviewing over in-person interviewing.

Of course, telephone interviewing is susceptible to other kinds of screening. In particular, screening resulting from use of telephone answering machines and from individuals who may refuse "by proxy" for the respondent (for example, a person answering the telephone refuses for the entire household). Recent research at our company suggests that answering machines are not a very significant source of nonresponse bias. In a recent statewide telephone survey in California, only about 4 percent of sampled observations could not be reached after 10 contact attempts because of the constant presence of an answering machine on the line. The proxy refusal and direct respondent refusal are far more serious problems, if only because most non-responses in telephone surveys arise in these categories.

While refusals have great potential to induce non-response bias, the mere fact that respondents refuse to participate in surveys is not necessarily evidence of the existence of nonresponse bias-except perhaps in extreme cases. To be sure, respondents do refuse to participate in surveys because of some aspect of survey content, which is likely to lead to nonresponse bias. However, they also refuse to participate for a large number of other reasons, most of which probably unsystematic and unrelated to the content of the survey. Most respondents appear to be reacting to the mode of the survey or to numerous other factors which are potentially unrelated to the content of the survey and thus are unlikely to produce non-response bias.

For example, in telephone surveys conducted at our firm about 60 percent of refusals occur before the appropriate respondent can be identified. That is, they occur during or immediately after the introduction to the survey. Moreover, in excess of 90 percent of refusals occur before the interview proceeds beyond the point of identifying the appropriate respondent. At this stage of the interview, the respondents have not really been exposed to the actual content of the survey so it can hardly be said that they are reacting to the content of the study-though it is clear they often are reacting to the mode of administration (that is, the cold telephone call).

Our interviewers, if possible, normally ask the respondent why they are refusing. (Interviewers seldom have time to ask this question, since most refusals at this point are rather definite and respondents usually hang up before the interviewer can speak.) Most respondents who answer this question indicate that they are too busy, that they are too tired, that they consider surveying to be an invasion of privacy or that they consider the survey to be an unwanted inconvenience. Few mention anything in relation to the content of the survey as their reason for refusing to participate. If we take these respondent at their word, it appears likely that the majority of refusals in telephone interviewing are probably unrelated to survey content and thus are unlikely to produce non-response bias. (Numerous surveyors have reported similar findings. See for example, "Nonresponse: The U.K. Experience," by Collins et al., in "Telephone Survey Methodology," by Groves et al., eds. John Wiley and Sons 1988.)

Of course, to the extent that the above reasons for refusing to participate in surveys tend to be geographically clustered, there may indeed be non-response bias induced by these differences. Urban populations, for example, are much more likely to refuse to participate in surveys citing the reasons outlined above. There is reason then in surveys which target urban and non-urban populations (for example, statewide surveys) to pay careful attention to the effects that differential response rates from these areas may have on survey results.

Controlling non-response bias-an overview

Except in relatively obvious cases such as those indicated above, researchers seldom know the extent of difference that may exist between respondents and non-respondents. Non-respondents by definition escape observation on most significant dimensions. Consequently, most efforts to control nonresponse bias focus on minimizing the survey's non-response rate and adjusting for non-response bias after the fact using analytical techniques when possible. (Efforts to analytically adjust survey estimates after the fact to take account of differences between respondents and non-respondents are not commonly used today, though more sophisticated survey designs sometimes anticipate the need for such adjustments and attempt to collect information that may be useful. In practice, survey designs involving such adjustments are difficult to explain and defend because so little is known about non-respondents.)

Another reason that control of non-response bias tends to focus on non-response rates is that clients tend to have an obsessive concern with these rates. Clients usually have strong opinions about whether a response rate is "good" or at least good enough, based either on their training or their prior experience with the population of interest of the subject under study. They tend to use survey response rate as a sort of catchall indicator of the quality of the survey effort; and surveyors who want to keep their clients are well advised to manage their client's perception of non-response rates carefully. It is often the only indicator that will be used to judge the quality of the survey work that has been undertaken.

Controlling non-response bias using CATI techniques

CATI offers a number of facilities that can be used to manage (though not eliminate) non-response bias at the survey and item levels. These include use of CATI systems to:

1. Cost-effectively enhance overall response rates for all kinds of surveys (mail, telephone and in-person);

2. Collect information to augment survey measurements (taken using mail or in-person survey techniques) for purposes of making statistical adjustments to population level estimates;

3. Measure the effects of non-response bias from mail and in-person survey techniques;

4. Collect survey information recursively collecting information for survey items that were previously not completed by the respondent (in mail or in-person surveys) - allowing researchers to eliminate existing item non-response bias; and,

5. Provide respondents with information that facilitates bounded recall-preventing the occurrence of item nonresponse bias.

Techniques one through four require either that the survey be conducted over the telephone or that telephone interviewing be integrated with other survey techniques such as mail and in-person interviewing. Use of a CATI system in telephone interviewing can greatly enhance the economic efficiency of interviewing, sample management and respondent data management-making possible the execution of mixed mode survey designs that would be otherwise prohibitively expensive to accomplish. However, a CATI system is not technically required in using the first four techniques. The last technique is virtually impossible to execute without a CATI system.

CATI as an integral tool in surveying

Unlike a watched pot, survey data have a tendency to fall painfully short of expectations if they are not continuously and closely inspected as they are collected. To ensure data quality, professionals who have a substantive understanding of the data being collected (consultants and project managers) should be able to routinely and easily inspect the operational results of surveying and analyze incoming data to identify problems that may be occurring. To facilitate this process, I believe the CATI system should be fully integrated with the other research facilities that may be part of the survey shop.

At FSC we conduct in-person, mail and telephone surveys. and various mixed mode versions of these surveys. To facilitate overall survey operations we have integrated our CATI facility with the other computer systems used by our consultants and project managers. We use a CATI/Ci2 (Ci2 System for Computer Interviewing) system in telephone surveying. The system runs on a dedicated 20-station computer network using Novell Netware v. 2.15. The CATI facility is connected to the 17-station front office network (also Novell Netware v. 2.15) using an internal bridge.

Because these systems are completely integrated, professionals working in the front office can easily attach to the CATI facility even when it is in operation. This allows them to:

inspect results of operations (such as completions and refusals)
observe interviews in progress (not very useful but it impresses clients)
analyze incoming telephone survey data quickly and efficiently, and
transfer sample management data to and from CATI/Ci2 (useful in mixed mode surveys and surveys using bounded recall).

Management of telephone survey response rates using CATI techniques

With the above operating design, it is possible for consultants or project managers to quickly analyze the operational or substantive results of surveying without leaving their desks. Depending on the professional's familiarity with CATI/Ci2 and other available software systems, he or she can analyze survey results using the "canned" facilities available in CATI/Ci2, or can load the data into one of the data base or statistical packages available on the front office computer network.

Consultants and project managers who are skilled in dBase have become adept at moving files back and forth between the call management data base (DB.CON) and dBase. This facility makes possible fairly in depth analyses of the results of survey operations. For example, it is possible to analyze response patterns by area code, telephone prefix, ZIP code, city and other geographic location information (if known). In practice, the professionals do not routinely use this facility to monitor non-response rates in the laboratory. Instead they tend to rely on the summary reports provided by the laboratory staff on a weekly or nightly basis. They also rely primarily on the laboratory supervisors to monitor the performance of interviewers and take corrective action as required. Analysts tend to analyze the call management database to answer unusual questions such as:

how successful have been efforts to date to convert initial refusals;
how long are the interviews taking and how much time on the average is being spent in the various stages of interviewing;
proportionately how many answering machines are being encountered and how many calls are typically required to "get around" them;
how are response rates varying by geographic location; and
in mixed mode surveying where we are telephone interviewing to follow up on unreturned mail surveys, how many of the respondents who are claiming to have sent in their surveys actually have done so.

In addition to analyzing the call management database, professionals also have the ability to rapidly load the results of interviewing into database and statistical packages such as SAS and SPSS. To facilitate this process, we have written specialized programs which parse the results of the Ci2 CONV2 program (the program that translates Ci2 results into ASCII format) and load them into dBase files. From dBase, these data can be analyzed directly or easily transferred to SAS or SPSS for subsequent analysis.

This facility is used in three ways. First, it is used to inspect early results from the survey. On more occasions than I like to admit, we have identified potential problems with question wording and logic by analyzing early survey returns. This facility provides us with the ability to do so. Clients also like to have preliminary results from telephone surveying. Often they are working under serious time pressure for preparing reports, and they like to use preliminary data to prepare analysis programs and begin "getting a feel" for the data that will eventually arrive. Finally, the facility is used in preparation of final survey deliverables.

Mixed mode surveying using CATI techniques

Mixed mode surveying offers a powerful approach to controlling and measuring non-response bias in surveying. By combining the various survey modes, it is possible to significantly reduce uncertainty about results due to the possible presence of non-response bias.

For example, mail surveys to businesses often produce relatively low response rates because the surveys are not addressed to a person in the organization who has the authority and responsibility for maintaining the information being sought. Response rates between 10 and 25 percent are quite common in this circumstance. It is difficult to have much confidence in survey results which contain such a large potential for non-response bias.

It is possible to significantly improve response rates to mail surveys of businesses by initially identifying the appropriate respondent in each business, through telephone interviewing. In this way, if the target of the survey is the purchasing manager, the survey gets delivered to the purchasing manager, who expects its arrival and has agreed to participate in the study. Using this two-stage survey technique, response rates on catical variables ranging from 65 to 80 percent are likely. By collecting basic information during the initial telephone interview that is critical for judging the eventual existence of non-response bias in the mail stage of the survey, it is possible to systematically study the presence of nonresponse bias and adjust for it if any is found. There is some additional cost involved, but few would argue that the improvement isn't worth it-especially if the validity of the data might eventually be challenged.

There are other mixed mode survey combinations that help to control non-response bias. These include:

telephone-mail-telephone designs-surveys which initially identify the respondent by telephone, send a self-administered instrument in the mail, and call back to the respondent to collect the required information;
mail with telephone follow up to non-respondents-the objective of the telephone follow up is to observe the differences between non-respondents to the mail survey and others who were willing to provide the information over the telephone. It doesn't completely eliminate non-response bias, but it can provide greater confidence in mail survey data and a means to adjust survey results to take account of nonresponse bias.

There are other mixed mode survey techniques which greatly reduce the cost of interviewing but have unknown impacts on non-response bias. For example, it is possible to combine telephone interviewing with in-person interviewing-using the former to identify and recruit respondents to the latter. The most commonly applied sample design used in in-person interviewing is the area probability sample usually with clustering. Surveys based on such designs are very difficult and expensive to carry out. If any selection criteria are applied within the sample (for example, sampling only for households with adolescent children), the costs of surveying using this technique may be prohibitive. Moreover, because of screening and other problems outlined above, these designs tend to be susceptible to serious non-response bias in urban populations. Telephone surveying to "recruit" respondents to in-person interviewing can greatly reduce the cost of in-person interviewing, and it is less susceptible to nonresponse bias due to screening and other biases. However, refusal rates on the telephone using this technique are quite high. For this reason, the jury is still out on this approach to surveying.

CATI techniques offer great improvements in efficiency over manual survey management approaches, in accompanying mixed mode surveys. Mixed-mode surveying typically requires that data be transferred either into or out of the telephone survey mode (for example, respondent name, address and other particulars that may be needed).

For example, in a telephone-mail survey, respondent contact information is typically collected during the telephone survey mode and used to address the mail survey mode. Other information obtained in the telephone mode may also be used to identify the appropriate mail survey version that will be sent to the respondent, if necessary. In this circumstance, it is possible to directly transfer respondent contact information from the CATI system into the mail processing system being used. This is particularly helpful in continuous surveying where results of one day's telephone interviewing "drive" the next day's mail survey batch. This approach also can be used to improve the quality of the mail survey materials. By loading respondent address information into word processing systems such as Microsoft Word or Word Perfect, it is possible to prepare highly personalized correspondence introducing the survey-an approach which has been shown to dramatically affect response rates.

Moreover, if the CATI system is tightly integrated with the other research facilities being used, it is possible to effectively track the status of sample observations as the survey proceeds. In mixed-mode surveying this can be a particularly difficult problem, because it is easy for respondents to get lost in the relatively automatic data collection process. Some of the clients we work for are extremely concerned about "bothering" the respondents (their customers). When respondents complain to the client that they don't want to be bothered anymore by the survey, it is necessary to exclude these respondents from further contact attempts. This can be easily done of the current status (mode and stage) of the respondent is known. If the CATI system is tightly integrated with the other tracking systems being used to track the status of cases, this is an easy problem which can be solved in a few minutes.

Using CATI techniques in studies using bounded recall

Finally, CATI systems are particularly useful in survey instrument designs which employ bounded recall as an approach to controlling both response and non-response bias. A major source of error in survey measurements is the respondent's memory. In some cases, the respondent's ability or inability to remember important details may significantly affect survey results. If respondents cannot remember details that are required to answer survey questions, they are less able to answer them. Moreover, respondents are more likely to refuse to continue participation when this occurs.

For example, in a recent study we performed for the Pacific Gas and Electric Company (PG&E) we were asked to determine whether or not a sample of their customers who had received advice concerning the costs and consequences of certain energy conservation options had implemented them. The recommendations were made to these customers from 1983 through 1989. The study was conducted in 1990.

Two separate "programs" were evaluated using similar surveys. However, in the case of one of the programs, detailed digital records had been maintained on the recommendations that had been made to each target respondent. With minor editing, it was possible to directly load the details of the recommendations and the consulting contact into the CATI/Ci2 system. Target respondents for this program were given advice between 1983 and 1986.

For the other program, which took place between 1987 and 1989, detailed information was not available concerning the recommendations that were made. All that was available were the name and address of the target respondent.

The difference between the response rates for the two studies were significant. For the earlier program, about 45 percent of respondents could be located and could remember receiving the information provided by PO&E. For the later program, about 31 percent of respondents could be located and remember receiving the information. Differences in recall rates were more dramatic. For the earlier program, about 80 percent of respondents could remember having received the specific recommendations (read to them by the interviewer) and could report what their organization had done about them. For the later program, only about 55 percent of respondents could remember specific recommendations well enough to be able to report how their organization had acted.

This example demonstrates that providing respondents with cues and other bounding information can significantly improve response rates, even for events and information which occurred quite some time ago.

Because surveys using bounded recall essentially involve a "unique" survey for each respondent, a CATI system is almost required to complete them. Features of the CATI which are critical to this kind of study are:

the ability to load digital information in the survey system from outside,
the ability to display it at the screen,
the ability to act on imported data logically for purposes of controlling the flow of the interview; and
the ability of randomly vary the order of presentation of questions, to control bias that might be induced by the ordering of questions presented at the screen.

Concluding remarks

CATI systems used in conjunction with survey techniques designed to control non-response bias can reduce problems associated with non-response bias. It seems to me that using a CATI system as I have described is one of the best available approaches to controlling such bias. This article has shown you some of the approaches that can be used.

However, non-response bias remains a sticky problem, which is not likely to go away any time soon. It is a problem which is probably getting worse in all modes of surveying. Since many readers are involved in telephone surveying, I would like to end with a few comments about what I think the major challenges are for controlling non-response bias in telephone surveying.

Telephone surveying using RDD sampling techniques offers an attractive sample frame and relatively inexpensive interviewing costs. However, there is one big problem with this approach to surveying-the rate at which respondents or their representatives refuse to participate.

To date, the most common approach to controlling refusal rates is to attempt to "convert" refusals. This approach is costly and typically doesn't yield much improvement in the refusal rate. In our experience, in the California market only about 15 percent of initial refusals can subsequently be persuaded to participate in the study. Considering that the average compliance rate for surveys in California is about 50 percent, this sort of improvement falls somewhat short of being impressive.

It seems to me that a significant methodological breakthrough will be required to really control the problem of refusal rates in telephone interviewing. We as surveyors probably need to shift our attention away from trying to shrink refusal rates to trying to measure or understand how those who refuse might be different from those who do not. Remember, it doesn't matter whether people refuse to participate in a survey if the probability of their having done so is unrelated to any of the measurements we are taking.

I think there are some ways of doing this which involve systematically studying refusals in an experimental fashion-but that is another topic.