Editor's note: Stefan Hartmann is principal of Sounding Weight Research, Seattle. He can be reached at stefan@soundingweight.com. This article appeared in the August 8, 2011, edition of Quirk's e-newsletter.
Finding respondents that match your target audience is critical when conducting any online survey. In most cases, especially when consumer research or business-to-business work among larger audiences is involved, finding the appropriate consumer group will entail working with a commercial research panel. Such panels offer quick and relatively inexpensive access to respondents, particularly in cases where the survey is a one-time effort.
But how can you as a client make sure the panelists are the people who can help you predict the success of your new idea? By asking lots of questions. Below are six key aspects to cover when trying to find the right online panel for your project.
1. How does the panel find respondents?
Locating respondents who are genuine, qualified and willing to share their opinions is a challenge since there are many platforms competing for those interested in taking surveys. And not all panel providers are equally careful in their recruitment efforts. So it is important to ask where the panel looks for new members: in high-traffic channels where lots of people congregate (such as Facebook or Google); in special-interest locations (such as professional or trade journals or Web sites for people who enjoy a particular activity), etc.
One increasingly common recruitment approach is called river sampling, where participants are recruited via online advertising, screened on basic demographic criteria and assigned to a survey on the spot if they qualify. In most cases, such river respondents (the term is derived from the river of online traffic) are not added to a panel's database, meaning it is often impossible to contact them after they have finished the survey.
All of these approaches have merit as long as the panel provider is transparent about its recruitment methods and screens new entrants carefully (see below). And as a client, you want to consider whether the panel's recruitment approach is a good way to find your audience and answer your research questions.
2. How are the candidates screened?
To get an idea about respondent quality, it helps to ask what types of questions potential participants need to answer before they can join a panel and what other checks are used to screen out dubious members. Does the panel make sure those listing a U.S. address do in fact access their surveys from an IP address located in the U.S.? Does the panel check that the responses are entered by a human being and not a computer program? Are participants checked for membership in other panels? What limits are imposed regarding multiple memberships? Does the provider examine whether the responses of the new panelist are actually plausible so that a person claiming to be a surgeon cannot report high school as his or her highest educational degree?
A wide variety of checks can be employed - as long as the panel has a consistent screening policy in place and is clear about how it works - that can satisfy your initial curiosity as a client. You can also try joining a panel yourself to experience the screening procedure from the respondent perspective.
3. How does the panel screen for response quality in your survey and what happens to problematic interviews?
No matter how carefully a panel initially screens its respondents, there will always be some that try to game the system - that's human nature. A good panel will make sure participants cannot take the same survey repeatedly and that those responding take the time to give answers that make sense. Ballot box-stuffing can be prevented by distributing unique links that can be used only once and it is also possible to block the survey from being taken more than once from the same computer. A variety of approaches are used for this so ask what the panel offers.
To control for quality of each interview, some panels automatically remove those who race through your survey (speeders); choose the same answer to all the statements they are asked to rate (straightliners); and those with generally problematic response patterns (high-school dropout with $250,000 annual income, really?). Some panels also add red-herring items ("Please check 2 in this list") to make sure respondents are paying attention. If not done automatically, a good panel will go through these steps at your request and should not charge you for interviews that seem suspect.
4. How does the panel compensate respondents?
The reward question is something that any client should ask about before deciding on a panel, since the response can indicate how much a panel cares about having satisfied members and keeping them on the panel. Good panels will offer some form of individual compensation for each completed survey. The amount may be modest (e.g., $2 for a short consumer survey) but even a small compensation will convey to respondents that their time and individual contribution are valued. For a business-to-business survey, much higher rewards are offered and they should be - after all, we researchers expect business-to-business respondents to draw upon professional knowledge for their answers. Some panels use point systems (similar to airline mileage programs) to reward respondents and credit a certain number of points for each completed survey. Once a certain number of points have been accumulated, panelists can exchange points for merchandise or gift cards. Since these programs require a little more commitment, they tend attract more valuable respondents.
Many panels consider the details of their compensation model to be part of their competitive advantage and therefore proprietary. But your contact at the panel should be able to give you at least an overview of how respondents are rewarded. Refusal to do so is a red flag.
5. How often are panelists contacted?
Good panels make sure that they reach a happy medium on the number of contacts - they don't overburden respondents with multiple invites over a short period of time but contact them regularly enough so members notice that the panel has surveys for them and stay active as members. A maximum of about two contacts a week is a good rule of thumb. On the other end, members should be invited to complete something at least once a month to keep them engaged. If there is no client survey to fill out, good panels will use this as an opportunity to conduct additional screening surveys for members, like asking which mobile device they currently use, to increase the chances that they will be needed for a future survey. As a client, you can also ask to get a question added to such a screening survey, if your specific audience is not pre-identified on panels yet.
6. Will the results represent your audience?
When you go forward with an online survey, you want to be sure that the results are representative of your target audience. Most panel providers will readily agree to this but it is worth asking what their commitment includes. Some panels will promise to make sure that the e-mails that are sent represent your audience. But as a client, you are more interested in making sure that the completed interviews match your target. There is sometimes a big difference between these two aspects, especially when the target audience includes groups with different response rates (e.g., young-adult consumers who tend to be less active respondents and their parents' generation, who are more willing to participate).
It is always worth asking what exactly is meant when a panel promises results that are representative to a particular target. Good panels will be able to give you a clear answer to this question.
Covered the basics
There are many more aspects to getting good responses to online studies, such as opt-out policies, software used, data security, survey design and length. But if you ask the six questions listed here and are satisfied with the answers, you have definitely covered the basics in finding a good panel.
Further reading
ESOMAR's 26 questions to help research buyers of online samples
CASRO and ESOMAR recently announced that they are working on a joint online sample quality standard focused on respondent authenticity, survey engagement, within-survey and cross-survey de-duplication and participation frequency. A first draft of the standards is expected later this year. Click here for more information.
Special thanks to Jonathan Tice of Authentic Response and Manfred Bluemel of Zeitgeist Research for providing valuable feedback and ideas for this article.