Five simple solutions to common telephone research problems

Editor's note: Eric DeRosia is assistant telephone center manager for Western Wats Center, Provo, Utah.

Data collection companies sometimes make mistakes that cause missed deadlines, higher costs, or even biased data. How can you, as a research professional, prevent data collection companies from making mistakes on your projects?

As an assistant telephone center manager for Western Wats Center, a data collection company in Provo, Utah, I have found that mistakes are often caused by five basic problems in the relationship between field service and client. Here are some simple ways you as a professional market researcher can solve these common problems and enjoy mistake-free data collection.

Problem #1

Poor communication between client undefiled service. Most data collection mistakes can be traced to poor communication. If project instructions are vague or difficult to understand, field personnel may misinterpret them and make mistakes. If project instructions do not cover the necessary topics, field supervisors may independently make crucial decisions that affect the outcome of the study--decisions better made by the client, who has the complete picture.

Miscommunication is possible in almost any situation. Consider the following example: Only a few hours before interviewing begins on a project the client contacts the field service and changes the instructions for one of the skip patterns. If the field service representative misunderstands the new instructions and there is no written verification of the changes, the entire project may be done incorrectly.

Yet another potential communication problem is a field service employee who has questions or foresees problems in the study but does not come forward with them because he or she is afraid to "bother" the client. There are some questions and problems that arise during data collection that require the immediate attention of the researcher. However, if the field personnel are apprehensive about telephoning the client at home at 9:30 p.m. to get instructions there is no communication link, and what was at one time a solvable problem may turn into an unnecessary tragedy.

Solution #1 - Write special field instructions. Field instructions should be written for the field personnel, explaining your expectations and instructions in detail. In this way, you can communicate directly with those actually administering your project. The field instructions should include the following:

1. Any anticipated administration difficulties including complex version splits, unusual sampling procedures, or complicated skip patterns.

2. General survey techniques you want to stress such as client- specific probing techniques.

3. Full instructions for briefing supervisors and interviewers.

4. Instructions on how to contact you if problems arise.

This last item is very important. Field personnel should be encouraged to contact you with any questions or problems that arise so that they will not be afraid to "bother" you with important questions and information. If possible, include your home telephone number and the latest time you are willing to be called. These questions from the trenches will often prevent major errors before they are made.

Field instructions may appear to require a great deal of effort, but after one set is written, subsequent field instructions will simply be modifications of the first. Thus, a great deal will be accomplished with a minimum amount of effort.

In addition, when verbal changes are made in a project, a quick fax verifying the changes will prevent misunderstandings.

Problem #2

Insufficient sample. Frequently, not enough telephone sample is generated for the project to be completed. For example, it is very difficult to finish a project when, even after following all possible call-back procedures, there are only 52 remaining "live" numbers to complete the final 47 surveys. Even if the data collection company detects this problem early, there still may not be enough time to generate more sample before the deadline. In this way, a sample shortages may make it impossible for the field service to meet the deadline.

Solution #2 - Generous amounts of sample. A generous amount of telephone sample should be created at the onset, avoiding unforeseen sample shortages. However, exactly what is "generous?" How much sample should be provided for a telephone research project?

According to Chris DeAngelis, account executive for Survey Sampling Inc. (SSI), the working phone rate, contact rate, cooperation rate, and project incidence should all be considered when calculating the optimum amount of telephone sample to be generated.

The first factor, working phone rate, is the percentage of telephone numbers in the sample that reach residential households (as opposed to disconnected numbers, businesses, etc.). Of course, this rate varies, depending on the method used to generate the sample. For example, one of the methods used by SSI to generate random digit dial (RDD) sample is two-digit randomization.

According to DeAngelis, SSI first stratifies RDD sample at the county level. Two random digits are then appended to the known universe of active area code, telephone prefix, and working block combinations. (A working block is the two digit combination which allows the prefix, e.g. the telephone number (203) 255-4200 falls in working block 42 of exchange 255.) With two random digits added, a ten digit (RDD telephone number is generated.

SSI has recently completed a study of his methodology, involving almost 100,000 dialing attempts. The results how that SSI's method yields a national working phone rate of 65%. adjusting this figure for non-English speaking and deaf households yields a rate of 56%. In other words, we can expect approximately 56% of numbers generated by this method to be English speaking households.

Of course, this rate varies with other sampling methodologies. DeAngelis estimates that randomizing the last four digits of actual telephone numbers yields a working phone rate of slightly lower than 25%. Sample from other sources, such as voting registration lists or customer lists, will typically have much higher working phone rates.

The second factor to consider when determining telephone sample size for a project is contact rate. This is a measure of the data collection company's ability to reach respondents at home. SSI has measured the national contact rate after three attempts to be 56.4%. Therefore, we can expect to reach a respondent in 56.4% of the households in the sample. This rate can be increased by attempting each telephone number more times, but this can require more time in the field, and is subject to the law of diminishing returns.

The third consideration is cooperation rate. This is the percentage of people willing to participate in the study. According to DeAngelis, the national cooperation rate is 53.2%. This rate will vary for different geographic regions, survey lengths, survey topic, etc.

Yet another consideration is the project incidence. This is, of course, the percentage of respondents who are eligible for the survey. Incidence rates usually range from approximately 75 %, for a project with few restrictions, to 5 % or lower, when there are many restrictions on those who qualify.

An example of how these rates work together may be helpful. Consider a set of 1000 telephone numbers. The working phone rate describes how many of these numbers will be households. If the working phone rate is 56%, then we can expect 560 numbers to be households. The contact rate then describes how many respondents we will contact by calling these households. If the contact rate is 56.4% after three attempts, we can expect to contact a person at only 56.4% or 316 of the 560 households we call. The cooperation rate describes how many of these contacts will be willing to participate. If the cooperation rate is 53.2%, we can expect that, of the 316 contacts we speak with, only 53.2% or 168 will participate. Next, we consider the incidence. If the incidence for the project is 75%, we can expect that only 75% or 126 of those willing to cooperate will be eligible for the survey.

This relationship is expressed in the following formula, which calculates the number of telephone numbers required per completed survey:

                                                       1                                         
(Working phone rare) (Contact rate) (Cooperation rate) (Incidence)

The optimum amount of sample is calculated by multiplying this ratio by the number of surveys to be completed, n.

Continuing our example of SSI's method of two-digit randomization, we can expect a working phone rate of 56%, a contact rate of 56.4%, and a cooperation rate of 53.2%. Using the above formula, we can calculate the amount of sample that should be generated for variations of incidence rates and sizes of n as shown in the table below.

Although this table applies only to SSI's two-digit RDD sampling method, it clearly shows a trend: studies with low incidence rates or large sample sizes require significantly more telephone sample.

In summary, the optimum amount of telephone sample for a project can be calculated by estimating the working phone rate, contact rate, cooperation rate, and incidence. Knowing the amount of sample a project will require will prevent you from generating too much (wasting money) or generating too little (making the project impossible to finish on time).

After the sample has been generated, field services can be required to use replicate sampling procedures as a method of insuring that proper callback procedures are followed. As part of this procedure, new replicates are distributed to interviewers only after all the preceding replicates have been exhausted. This allows a large amount of sample to be distributed in an even and controlled manner.

A simplified version of replicate sampling is acceptable in many situations. This involves dividing the sample into two large sets. The first should be approximately sufficient to finish the project. The second and smaller set should not be opened at all, unless the first set is completely exhausted and the client has given approval. In this way, a "cushion" is provided, preventing sample shortages. If the sample is costly and a generous amount cannot be provided, a daily sample disposition can be requested from the field service, listing the total number of "live" and "dead" telephone numbers in the sample. Data collection companies sometimes charge a small fee to gather this information, but it will give you advance warning of any problem sample shortages.

Finally, a contingency plan for sample shortages can be worked out in advance. If this is done, a phone call from your data collection company with bad sample news can be handled quickly and easily.

Problem #3

Difficult to understand skip pattern instructions. Sometimes interviewers are able to grasp difficult survey instructions during the briefing, but consistently make mistakes on those instructions during actual interviewing. These mistakes will be discovered during the editing process and corrected, but is that acceptable? For example, imagine a difficult skip pattern that causes many interviewers to mistakenly skip an unaided awareness question. When the mistakes are discovered by editors, each respondent must be called back and asked the skipped question. At this point, however, all the respondents have heard the entire survey, making their responses to the un-aided questions biased. This process effectively changes the order the questions are asked. If the survey's instructions are too difficult to follow during interviewing, many surveys will have to be corrected in this way, biasing the data for some question types.

Solution #3 - Write very simple and clear instructions on skip patterns. Skip pattern instructions should be worded as simply as possible so interviewers can follow them easily.

A poor example:

12) Do you support or oppose the initiative?

14) ASK ONLY IF "Oppose" IN Q. 12: Why do you oppose the initiative?

A better example:

12) Do you support or oppose the initiative?

13) * Why do you support the initiative?

14) ** Why do you oppose the initiative?

The second example will need to be explained during the briefing, to ensure that interviewers ask only one of the two open-ended questions and not both, but it is much easier for interviewers to follow during an actual interview than the first example. This format of highlighting the questions that should be asked with asterisks will not work in all situations, but it is an example of the level of simplicity that is required for easy interviewing.

Unavoidably difficult skip patterns should have a detailed explanation in the field instructions. If the survey is too complex, CATI applications can be used, preventing interviewing errors, editing errors, and eliminating data entry costs.

Problem #4 

Unclear screening questions. When a survey is administered, the interviewers will contact thousands of respondents, people in every imaginable situation. If the survey instructions do not contain a clear definition of the target population, questions will arise about the eligibility of respondents.

For example, if a data collection company begins to administer a survey, and the instructions have an age requirement, interviewers will inevitably reach a seventeen-year-old head-of-household and ask the supervisor if that person is eligible. At that point, the supervisor may make a wrong decision. If the supervisor mistakenly allows people under 18 to be interviewed, data will be collected from respondents outside the target population, with no way to later differentiate between data that should have been included and data that should not have been included. Although good data collection companies have established procedures for dealing with this common problem, relying on the frontline supervisor to act correctly can be risky.

Solution #4 -Define the target population in the instructions very clearly. By taking all possibilities into account when designing the screening questions and including a clear definition of the target population in the survey instructions, interviewers and supervisors can make correct decisions regarding the eligibility of respondents.

In addition, a pre-test is very useful for flushing out possible problems or loopholes in the screening questions.

Problem #5

Misplaced interviewer instructions. If interviewer instructions such as [DO NOT READ CHOICES], [ROTATE], or skip patterns are not in an obvious place on the page, interviewers may overlook them. If interviewers consistently fail to follow any of these instructions, the collected data may be biased.

Solution #5 - Do page layout with interviewers in mind. Interviewer instructions are easiest to see when they are in the interviewer's line of sight, immediately following what they have just read or immediately preceding what they are about to read.

A poor example:

12) Who initiated the bill in congress?

A better example:

12) Who initiated the bill in congress? (DO NOT READ CHOICES)

When asking the first question, interviewers will often mistakenly read the choices, while the second question will be asked correctly almost every time.

Of course, it is the responsibility of the data collection employees to administer projects accurately. Experienced and reputable field services take this responsibility seriously. However, even with thorough training and the best quality control systems, people will still make mistakes.

As a research professional, you can play a large role in preventing these mistakes from being made on your projects. Ed Ledek, client service vice-president for Western Wats Center, explains the role of professional researchers in this way: "We have to work closely with our clients in order to avoid errors and mistakes. When we understand exactly what the client wants, our people can successfully make all the little judgments that go into a project."

Special field instructions, generous amounts of sample, clear skip patterns, specific screening question instructions, and easy to follow page layout will all help prevent mistakes during the data collection phase of your telephone research projects.