By the Numbers: Are you surveying the right professionals?

Multimillion-dollar decisions hinge on the results of market research. The stakes are so high for corporations, it is vitally important the correct respondents are interviewed. Interviewing qualified respondents makes all the difference between gathering useful data and useless data, between your company making money or losing money, between career advancement or something else!

The problem is, potential respondents also have dollars to gain - incentive dollars. These incentives motivate some respondents to cheat, to try to qualify for studies they should not gain access to. Since business professionals earn higher incentives than an average consumer, more people try to sneak into studies geared for business professionals. How happy would you be if you found out that a clever college student collected a $100 incentive to give his opinion on a new heart valve? Or a high-school senior provided his feedback on the relative importance of features in corporate enterprise security software and walked away with $50 in beer money? Here are some proven methods to help remove the wrong kind of professional respondents from your study.

First: Be careful about how much information you provide regarding the study’s topic.

Too much detail allows the “professional” respondents to maneuver their way through a screener. If your introduction states that your study is about how people use their MP3 players, then respondents know they need to check “yes” when asked whether they own an MP3 player. The caveat is that some general information on the study topic needs to be provided. Otherwise some people will not agree to the study because they don’t know what they are getting themselves into. An alternative introduction could indicate that you are interested in people’s habits when listening to music.

Second: Focus closely on the screener questions.

The purpose of a screener is to accurately identify the correct respondent and screen out unqualified respondents. However, not all screeners are created equal. Here are four key characteristics that good screeners follow:

1. Avoid using yes/no questions: A “professional” respondent has a 50 percent chance of choosing correctly if the question only has two choices.

2. Be cautious of leading questions: Sometimes the question wording can leave clues as to what the researcher is looking for. Instead of asking “Do you sell notebook computers?” ask “What type of computers do you sell?” The response list should include desired responses as well as undesired responses. This way the “professional” respondent will have a lower chance of selecting the right answer.

3. When practical, combine demographic profiling questions with occupational questions to screen out illogical answers and filter out unqualified respondents. For example, it is very unlikely that a neurosurgeon has less than a certain number of years of education and must be at least a certain age. If answers to age and years of education do not match with job descriptions, something is probably wrong. There are not many 19-year-old neurosurgeons.

4. Do not terminate any respondent until all screening questions have been answered. If a respondent is terminated on the question which disqualified him, he may be able to sneak back into the survey and choose a different answer. This process could potentially be repeated until the “professional” respondent gets through the whole screener. This is possible because some respondents have joined Web panels using multiple e-mail addresses.

Third: Monitor the incentive amount.

The higher the incentive, the more effort a “professional” respondent will go through to be involved in a study he is not qualified for. Still, you do not want to underpay. Consider a respondent’s time and profession when determining an appropriate incentive. There are expected levels of incentives for certain professions and paying below these standards could lead to dissatisfaction among qualified respondents or higher costs in recruiting. It is also a good idea to stress the value of participation. This encourages responses motivated by altruistic desires rather than purely economic incentives or greed.

Incentives will vary by country, especially since the legalities of incentives vary by country. Also, incentives may not be necessary for some surveys. If you are surveying from a panel, the panel company should have a system in place for paying its members.

Gaming the system

“Professional” respondents know they have an opportunity to make a lot of money and win the challenge of gaming the system. The problem is widespread across panel communities and panel companies are taking drastic, aggressive measures to curb this problem. Common approaches are phone validation, allowing only one panelist per home address, etc. Approaches such as this are broad-brush attempts to solve the problem. “Professional” respondents are resourceful, and their innovations to gain illegitimate access to your surveys are constantly evolving.

It is the researcher’s job to insure the integrity of the data collected within a survey by insuring the respondents entering and completing your survey are qualified to do so.

Remember, most surveys are custom and with that comes custom approaches to keeping unqualified respondents out. The goal is to eliminate unqualified respondents without making the survey (or the screener) too cumbersome for the qualified respondents whose feedback is so critical to your organization’s success.