Addressing a growing problem

Editor’s note: Mansour Fahimi is vice president, statistical research services, at Marketing Systems Group, a Fort Washington, Pa., research firm. Dale W. Kulp is president and CEO of Marketing Systems Group.

Increasingly, survey and market researchers are reconsidering address-based sampling (ABS) methodologies to reach the general public for data collection and related commercial applications. Essentially, there are three main factors for this change:

•  evolving coverage problems associated with telephone-based sampling methods;

•  eroding rates of response to telephone contacts along with the increasing costs of remedial measures to counter nonresponse; and on the other hand

•  recent improvements in the databases of household addresses available to researchers.

In this article, we provide an assessment of these three factors, evaluate pros and cons of ABS as an alternative, and discuss specific enhancements that can establish this emerging methodology as a practical solution for market research applications. In particular, such enhancements include amelioration of some of the known coverage problems associated with ABS frames as well as their augmentations with demographic, geographic and other supplementary data items. While reducing bias due to undercoverage, such enhancements enable researchers to develop more efficient sample designs as well as broaden their analytical possibilities through an expanded set of covariates for hypothesis testing and statistical modeling tasks.

Fundamental changes

For the past decade and a half, a large portion of telephone surveys have been based on the random-digit dialing (RDD) methodology where telephone samples are generated within the 100-series telephone banks that contain at least one listed number. During the intervening years, this method of list-assisted RDD has overlooked the many fundamental changes in the U.S. telephony and relied on a convenient assumption that elimination of other telephone banks from the sampling frame amounts to exclusion of a small percentage of households, hence resulting in an ignorable coverage bias.

However, recent investigations suggest that the extent of this coverage bias is now approaching 20 percent and growing (Fahimi et al. 2008). More specifically, as seen in Table 1, about 15 percent of this undercoverage is attributed to residences whose telephone numbers are now in excluded telephone banks with no listed numbers while the remaining 5 percent are in POTS exchanges with mixed-use banks with no listed numbers. Moreover, with a growing number of cell-only and cell-mostly households, it is estimated that three out of every 10 adults receive all or nearly all of their calls on cell phones. Put together, the conventional RDD samples can fail to cover over 40 percent of the households - a problem that becomes even more pronounced when surveys target special subpopulations such as younger adults.

Biener et al. (2004) and Curtin et al. (2005) point out that the rate of response to telephone surveys has been on a decline. More recent investigations by Fahimi et al. (2007a) suggest that the national rates of response to the Behavioral Risk Factor Surveillance System (BRFSS) survey, which is the largest RDD survey in the world, follow this trend as well. As shown in Figure 1, BRFSS has suffered a drop of nearly 20 percentage points in response rates during the course of the past decade.

Given that nonresponse is highly differential in nature and varies significantly across different demographic subgroups, it is of a great concern when over half of the sample households opt not to respond to a survey. Even when sophisticated nonresponse adjustment procedures are employed to reduce the incurred bias, it would be farfetched to assume such remedial procedures can reduce nonresponse bias to a tolerable and measurable level. Also, it should be noted that reducing nonresponse bias via weighting is always exercised at the expense of the precision of survey estimates, since weight adjustments inflate variance estimates (Fahimi et al., 2007b).

Beyond statistical techniques, many researchers have resorted to other tactics to improve response rates to surveys. As reported by Fahimi et al. (2004) the offer of incentives can significantly increase response rates, however, even an increase of 10 to 20 percentage points can still leave a survey with an overall response rate below 50 percent. Moreover, marginal gains in response rate are often achieved at a high cost, as practical nonresponse conversion strategies are labor intensive and require exceedingly larger amounts of incentives to be effective. Coupled with the non-monetary cost due to loss of precision mentioned above, the overall cost of dealing with nonresponse can be prohibitive.

Promising alternative

Recent advances in database technologies along with improvements in coverage of household addresses have provided a promising alternative for surveys and other commercial applications that require contacts with representative samples of households. Obviously, each household has an address and virtually all households receive mail from the U.S. Postal Service (USPS). The delivery sequence file (DSF) of the USPS is a computerized database that contains all delivery point addresses, with the exception of general delivery where carrier route or P.O. box delivery is not available and mail is held at a main post office for claim by recipients. The second generation of this database (DSF2) is the most complete address database available. With more than 135 million addresses on file, it is safe to assume that if an address cannot be matched against DSF2 it is probably undeliverable. By providing validation services for both correctness and completeness of addresses, DSF2 can significantly enhance the address hygiene. Consequently, this system helps reduce the number of undeliverable-as-addressed mailings, increase the speed of delivery and reduce cost. Also, with daily feedback from tens of thousands of letter carriers, the database is updated on a nearly continuous basis.

Given the evolving problems associated with telephone-based methods of sampling and data collection, many researchers are considering the use of DSF for sampling purposes. Moreover, the growing problem of nonresponse - which is not unique to any individual mode of survey administration or country (de Leeuw and de Heer 2002) - suggests that more innovative approaches will be necessary to improve survey participation. These are among the reasons why multimode methods for data collection are gaining increasing popularity among survey and market researchers. It is in this context that ABS designs provide a convenient framework for an effective administration of surveys that employ multimode alternatives for data collection.

Different strategies

Considering that through reverse-matching the telephone numbers for many addresses can be obtained, different strategies for a multimode survey administration can be developed to accommodate the timing, budgetary and response rate needs of a survey. One such strategy could start with the selection of a DSF-based probability sample of households in the geographic domain of interest. This sample may be selected across the entire domain, or clustered in an area probability fashion if in-person attempts are contemplated as part of the design. Initial contacts can be by phone and/or mail and can include attempts for survey administration at the same time. Alternatively, this first contact can serve as a recruitment effort to invite potential respondents to participate in the survey via Web, dial-in numbers for live interviewing, an IVR system or other options. Once the nexus of contact modes has been developed for each respondent, further contacts and reminders for survey completion can take place in any order or combination of modes that meets the project needs.

Cognizant of the potential implications of combining different modes of data collection, the emerging conclusions from many studies seem to suggest that different research modalities can often be combined effectively to boost response rates (Gary 2003). In comparison to an RDD-only approach, in particular, an address-based design using multiple modes for data collection can provide response rate improvements, cost savings, as well as better coverage for households that are completely uncovered by landlines (Link 2006). As for comparisons with in-person and mail-only modes of data collection, needless to say, the former is too costly to be practical for many applications while the latter (with notoriously low rates of response) requires expensive nonresponse follow-up efforts to produce creditable data (Groves 2005). What seems critical, however, is for researchers to minimize differences between survey instruments associated with each mode. Moreover, effective weight adjustment techniques might be needed post-data collection to account for the observed differences in the profile of respondents to each mode.

Higher likelihood

As reported by a number of researchers, certain households have a higher likelihood of not being included as a delivery point on the DSF. Staab and Iannacchione (2003) estimate that approximately 97 percent of all U.S. households have locatable mailing addresses, however, this prevalence diminishes with population density and approaches zero in areas where home delivery of mail is unavailable. Dohrmann and Mohadjer (2006) report that when comparing lists of on-site-enumerated addresses to DSF generated listings of households for the same geography, in rural areas the rate of mismatches can be over 23 percent. However, these researchers do indicate that as rural area addresses go through the 9-1-1 address conversion and acquire a city-style format, the coverage of DSF-based lists in rural areas is likely to improve in the future. Also, O’Muircheartaigh et al. (2003) point out another source of undercoverage for address-based samples when lists of households are purchased directly from commercial list compilers because households can request that their addresses not be sold.

Beyond coverage issues, when DSF-generated samples are used in surveys that adopt a multimode approach for data collection one has to be prepared to address concerns about mode effects. While somewhat academic in nature, concerns have been raised about systematic differences that can be observed when collecting similar data using different modes (Dillman 1996). On the one hand, several studies have shown a greater likelihood for respondents to give socially-desirable responses to sensitive questions in interviewer-administered surveys than in self-administered surveys (Aquilino 1994). On the other hand, the rate of missing data is often significantly higher in self-administered (mail or Web) surveys as compared to interviewer-administered (telephone or in-person) surveys (Biemer et al., 2003).

While roots of differences in data quality and response rates between various modes of data collection deserve further investigations, some solace may result when surveys are administered without confining data collection to any single mode. Arguably, certain shortfalls of one method might be mitigated when other methods of data collection are made available to the respondents as well. Ultimately, however, it might be impossible to untangle the immeasurable interactions between the mode, the interviewer, the respondent and the survey content (Voogt and Saris 2005).

Comprehensive sampling frame

As mentioned above, the current version of DSF can serve as a comprehensive sampling frame for address-based survey applications when complete coverage and proper representation of the target population are among the nonnegotiable features of the sample design. In addition to a near-perfect coverage of the entire country, this versatile database can provide scientific samples for finely-defined geographic subdomains - a resilient point of contention for telephone-based surveys, particularly in light of the growing coverage problem resulting from landline number portability. In addition to the available information that can be accessed directly from DSF2, it is possible to append many ancillary data items to each address for use in complex sample designs that require detailed information for stratification purposes. This is the crossroad where basic list suppliers, those that simply offer extracts from what the USPS provides, are differentiated from statistical sampling companies that provide enhanced versions of DSF. Key enhancements of DSF include: detailed geodemographic information; name and telephone number retrieval; and simplified address resolutions.

Detailed geodemographic information. While DSF can provide basic delivery details about an address, oftentimes researchers require detailed geodemographic data for sample design and allocation. By accessing several commercial databases that contain various data items for households, it is possible to enhance DSF for targeted sampling applications. While many of such data items correspond to individual households, there are also modeled characteristics that are available at different levels of aggregation. Starting from the ZIP+4 level, which typically consists of only a handful of households, the resulting information can then be rolled up to higher levels, including all Census geographic domains (block, block group, tract, county, MSA, state and region); marketing geographic domains (media markets, zip areas, etc.); as well as custom areas (retail trading areas and specific geographies based on distance or radius).

Name and telephone number retrieval. Customizing the initial mailings to sample households is known to improve response rates and reduce cost. Given the plethora of junk-mail that households receive on a daily basis where the packets typically carry generic contact names, research suggests that the rate of response can increase significantly when the name of survey recipients appear on the mailed material (Dillman 1991). Moreover, with multimode survey applications one can reduce the number of nonrespondents to the mail survey through follow-up phone calls. Taking advantage of databases such as infoUSA, Acxiom and Experian, it is possible to retrieve names and telephone numbers associated with many of the DSF addresses. Done correctly, about 85 percent of addresses can be name-matched and over 60 percent can be linked to a landline telephone number - match rates decrease with inclusion of P.O. box addresses.

Simplified address resolutions. Since DSF only provides counts of undeliverable (simplified) addresses that are void of street numbers or other pertinent delivery information, resolution of such cases is among the most important enhancements that can be added when selecting address-based samples. While the number of such addresses is rapidly decreasing as they go through the 9-1-1 address conversion, currently there are about 1.5 million simplified addresses in the DSF. As seen in Figure 2, the distribution of simplified addresses varies across states, with West Virginia topping the rank with more than 30 percent of its addresses considered to be simplified. Again, by accessing several large databases that contain different information for households, one can obtain the missing information for virtually all simplified addresses. Subsequent to this resolution, all other informational data that exist for addressed households become available for sample design and data collection purposes.

Participation difficulties

All single-mode methods of data collection are subject to a growing range of coverage and participation difficulties on varying bases. Telephone-based surveys suffer from both coverage and response rate problems; in-person interviews are typically too costly to be practical as the only mode of data collection in many instances; and mail surveys alone often secure too low of a response rate to produce reliable results.

It is against this background that multimode methods of data collection are gaining popularity as alternatives that can reduce some of the problems associated with single-mode methods. As such, address-based samples provide a convenient framework for effective design and implementation of surveys that employ multimode alternatives for data collection. In this regard, the delivery sequence file of the USPS - once properly enhanced and prepared - provides a powerful tool for sample surveys. Enhancements provided by reputable organizations can improve the coverage of DSF and expand its utility for design and analytical applications.

References

Aquilino, W.S. (1994). Interview mode effects in surveys of drug and alcohol use: a field experiment. Public Opinion Quarterly , 58, 210-40.

Biener, L., Garrett, C.A., Gilpin, E.A., Roman, A.M., and Currivan, D.B. (2004). Consequences of declining survey response rates for smoking prevalence estimates. American Journal of Preventive Medicine , 27(3), 254-257.

Biemer, P.P. and Lyberg, L.E. (2003). Introduction to Survey Quality , New York: John Wiley and Sons.

Blumberg, S. J. and Luke, V. J. (2007). “Wireless Substitution: Early Release of Estimates from the National Health Interview Survey.” http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless200805.htm.

Brick, J. M., J. Waksberg, D. Kulp, and A. Starer. 1995. “Bias in List-Assisted Telephone Samples.” Public Opinion Quarterly , 59: 218-235.

Curtin, R., Presser, S., and Singer, E. (2005). Changes in telephone survey nonresponse over the past quarter century. Public Opinion Quarterly , 69, 87-98.

de Leeuw, E. and de Heer, W. (2002). Trends in household survey nonresponse: a longitudinal and international comparison. In R. M. Groves, D. A. Dillman, J. L. Eltinge (Eds.), Survey Nonresponse (pp. 41-54). New York: John Wiley and Sons.

Dillman, D. A. 1991. “The Design and Administration of Mail Surveys,” Annual Review of Sociology , 17, 225-249.

Dillman, D., Sangster, R., Tanari, J., and Rockwood, T. (1996). Understanding differences in people’s answers to telephone and mail surveys. In Braverman, M.T. and Slater J.K. (eds.), New Directions for Evaluation Series: Advances in Survey Research . San Francisco: Jossey-Bass.

Dohrmann, S., Han, D. and Mohadjer, L. (2006). Residential Address Lists vs. Traditional Listing: Enumerating Households and Group Quarters. Proceedings of the American Statistical Association, Survey Methodology Section, Seattle. pp. 2959- 2964.

Groves, R.M. (2005). Survey Errors and Survey Costs , New York: John Wiley and Sons Inc.

Fahimi, M., M. W. Link, D. Schwartz, P. Levy and A. Mokdad (2008). “Tracking Chronic Disease and Risk Behavior Prevalence as Survey Participation Declines: Statistics from the Behavioral Risk Factor Surveillance System and Other National Surveys.” Preventing Chronic Disease (PCD), Volume 5: No. 3.

Fahimi, M., D. Creel, P. Siegel, M. Westlake, R. Johnson, and J. Chromy (2007b). “Optimal Number of Replicates for Variance Estimation.” Third International Conference on Establishment Surveys (ICES-III), Montreal, Canada.

Fahimi, M., Chromy J., Whitmore W., and Cahalan M. Efficacy of Incentives in Increasing Response Rates. (2004). Proceedings of the Sixth International Conference on Social Science Methodology. Amsterdam, Netherlands.

Fahimi, M., Kulp, D. and Brick, J. M. (2008). Bias in List-Assisted 100-Series RDD Sampling. Survey Practice, September 2008, http://surveypractice.org/2008/09/.

Gary, S. (2003). Is it Safe to Combine Methodologies in Survey Research? MORI Research Technical Report.

Iannacchione, V., Staab, J., and Redden, D. (2003). Evaluating the use of residential mailing addresses in a metropolitan household survey. Public Opinion Quarterly , 76:202-210.

Link, M., M. Battaglia, M. Frankel, L. Osborn, and A. Mokdad. (2006). Addressed-based versus Random-Digit-Dial Surveys: Comparison of Key Health and Risk Indicators. American Journal of Epidemiology , 164, 1019-1025.

O’Muircheartaigh, C., Eckman, S., and Weiss, C. (2003). Traditional and enhanced field listing for probability sampling. Proceedings of the American Statistical Association, Survey Methodology Section (CD-ROM), Alexandria, pp. 2563-2567.

Staab, J.M., and Iannacchione, V.G. (2004). Evaluating the use of residential mailing addresses in a national household survey. Proceedings of the American Statistical Association, Survey Methodology Section (CD-ROM), Alexandria, pp. 4028-4033.

Voogt, R. and Saris, W. (2005). Mixed mode designs: finding the balance between nonresponse bias and mode effects. Journal of Official Statistics . 21, 367-387.

Wilson, C., Wright, D., Barton, T. and Guerino, P. (2005). “Data Quality Issues in a Multi-mode Survey” Paper presented at the Annual Meeting of the American Association for Public Opinion Research, Miami.