A look at some factors that affect telephone data collection costs

Incidence and over-quotas

Editor’s note: Takako Komatsu is operations analyst with Communications Center, Inc., a Washington, D.C., data collection firm.

In addition to choosing the right research methodology, today’s market researcher also must make sure that a project meets their budgetary goals. The fielding of a study with a telephone data collection firm may be one of the costliest portions of such a project. However, it is often very difficult for the phone center to provide an accurate estimate of the expenditures and resources necessary for the project before it starts. Some examples of factors that can affect data collection costs: the incidence of qualified respondents may be different than expected; the sample may not be as productive as expected; the study may require a certain number of data or points from a low-incidence respondent type.

In this article, we will quantify some common issues encountered by telephone data collection firms in estimating the resources necessary to complete a project. The concept of incidence and population will be redefined in the context of telephone data collection. This discussion will be extended to demonstrate how it can be used to estimate the impact of closing quotas.

Qualification

One of the most crucial pieces of information in estimating costs for a project in telephone data collection is the incidence of qualification. In general research, “population” refers to the group of people in the universe to which the researcher wants to generalize the conclusions of the study. At our firm we have found that in predicting interviewer hours, it is more practical and beneficial to think of population in terms of the universe of phone numbers (sample). By redefining what we think of as the population to fit in to the context of telephone data collection, we can arrive at a more accurate estimate of incidence which is crucial in predicting the resources necessary to complete a project.

For example, in a consumer satisfaction study about a product, we may know that 20 percent of households in the U.S. use this particular product. This may or may not be relevant in predicting resources needed to collect sufficient data. When estimating incidence of qualification we must always take into account the type of sample that will be used along with the qualifying characteristics of the respondent. This incidence of 20 percent, based on the product usage in the general population, is accurate only if we are calling from a untargeted, computer-generated list of phone numbers. If a list of names and phone numbers of registered customers is going to be provided for this study we need to use the percentage of working numbers that lead to an actual customer within the given phone number list (sample) as incidence. In this case, non-qualified respondents occur only when the sample leads to a household where the target respondent cannot be reached (e.g., they have moved away), or if the respondent claims that he or she is not a customer.

Moreover, using a phone number list as our population base is especially crucial when we use outside sources such as census data to estimate incidence. For example, the population ratio of Hispanics is 11.66 percent of the total population. This incidence may not be an accurate representation for telephone data collection because the total percentage of Hispanic households within all households in the U.S. is 8.40 percent (percentages based on census data 1998). When we are calling to reach Hispanic respondents, “household” units become a closer representation of the population of phone numbers with which we should base our estimations. Hence the Hispanic household ratio of 8.40 percent will probably be a closer estimate of the actual incidence observed in the study.

Significant impact

Quota requirements also have a significant impact on the amount of resources necessary to complete data collection. By quotas we mean the target number of completed surveys based on respondents’ characteristics (e.g., gender, age, or degree of product usage). Target counts, or quotas, are set before the start of the project to ensure that data is collected according to needs of the researcher. A study may require data to be collected in higher proportions of one group because the other groups are needed only as a baseline, hence requiring less data from the latter group. Or a researcher may need more data from a low-incidence group to increase the power of statistical analysis. Hence target quota counts may or may not reflect the actual incidence of that type of respondent in the population. Any researcher involved in the actual field operation of telephone data collection is aware that closing quotas can have a large impact on the cost of a project. When the discrepancy between quota ratios and population ratios is large enough, estimating the necessary interviewing hours for a project can become extremely difficult and unpredictable.

To demonstrate the impact of quota requirements on total interviewing hours, consider the following hypothetical projects. Study 1 requires a total of 200 surveys, 100 males and 100 females, each divided for three ethnic backgrounds (60 Caucasian, 30 African-American, 10 Hispanic). Study 2 requires a total of 200 surveys, 100 males and 100 females. But they must also have at least 33 respondents in each of the three ethnic backgrounds. We can easily see that in this example Study 2 will be considerably more difficult to complete compared to Study 1. Because the occurrence of Hispanic respondents is considerably lower than either Caucasians or African-Americans, it is most likely that we will inadvertently start an interview with them only to terminate them in order to meet quota requirements for Hispanics. The Caucasian or African-American respondents we call after we have already collected enough data from those categories are terminated as “over-quota terminates.”

If we have to terminate a large number of respondents due to over-quotas, it will have a significant impact on the resources necessary to complete data collection on a project. This impact needs to be quantified before the project starts fielding. These terminates are not due to non-qualification or from a lack of cooperation from the respondent. They are fundamentally different from screener terminates and cannot be categorized as unqualified respondents. We need a way to take this factor into account independently of the overall incidence of the project.

At Communications Center we are using a factor called “Over Quota Ratio” to accurately estimate the impact of closing quotas and over-quota terminates on the resources necessary to complete a project. The Over Quota Ratio refers to the average number of over-quota terminates that will be incurred per completed interview by the end of the project within a given incidence and quota requirement. This ratio is then used to adjust the interviewing hours necessary to complete the project under the quota requirements.

To do this at the time of project setup, we need to know two things:

1) The incidence of the all quota criteria occurring within the population (sample).

2) The exact quota counts or quota ratios that the client would like represented in the final data set.

Take for example a project that has gender quotas of 50 males, 50 females. We would expect some female OQ because of the disproportionate number of females who answer the phone. We have found that in telephone interviewing the population answering the phone is roughly 40 percent males, 60 percent females. After 100 completed contacts, there would be 40 males and 60 females, already yielding 10 over-quota females. In order to complete the project there will be an additional 15 over-quota females. At the end of the project we will have incurred 25 over-quotas and an OQ Ratio of 25 percent. From this we know that this project will need 25 percent more resources (interviewing hours) to complete compared to a project with no quotas or with quota requirements that more closely reflect the population incidence.

Accounting for over-quota terminates is particularly important because of the changing dynamics of telephone data collection. It gets more difficult to reach a live person every year. Anytime a qualified respondent is terminated due to closing quotas, the hours necessary to complete the project increase exponentially. Usually these quota questions are placed at the beginning of the survey along with the screener questions to minimize the time an interviewer spends talking. However, we have found that minutes saved from terminating over-quota respondents are insignificant compared to the amount of resources necessary to reach the next prospective respondent. Except in cases of very long surveys, the time incurred in over-quota terminates is approximately the same as that incurred in completing an entire survey.

Obvious merits

In the context of telephone interviewing, our firm has found that there are obvious merits in making assumptions about the population based on the phone number list used for the project. It guides us in how to use census data and other sources to determine incidence and to predict project hours. It also helps us understand and estimate the impact of quota requirements on the resources necessary for a project. In so doing, it enables us to provide more accurate cost estimates and advise our clients on how the project can be optimized for their data collection needs and budget. Further, we can provide options on how cost may reduced by adjusting quota requirements, changing screener questions, purchasing targeted sample or employing other methods.