Keep your house in order

Editor’s note: Jeffrey Henning is founder and vice president of strategy at Vovici, a Rockland, Md., research firm.

Market researchers often complain about the quality of online panels, whose problems have been well documented by the American Association for Public Opinion Researchers, the Advertising Research Foundation, ESOMAR and other industry groups. They are unrepresentative of consumers in general, often produce dramatically different results at different times and suffer from respondents who speed through surveys or even lie when answering questions.

Just as important, but much less talked about, is the quality of house panels.

House panels?   A house panel is a centralized resource, built on the foundation of customer lists but maintained specifically for the purpose of conducting survey research. Panel data can be maintained manually, through custom databases or through panel-management applications.

Since proprietary panels are built for internal use, they are not subject to the scrutiny that comes when an access panel markets and sells its capabilities. For instance, the team managing a house panel never publishes the answers to the ESOMAR 26, which is a staple of commercial panels, addressing the 26 questions buyers should ask panel providers during the sales process. This lack of outside scrutiny is typically not replaced by a layer of inside scrutiny either. Often, organizations haven’t even documented their own practices for new members of their team. Firms are often unaware that they are making mistakes in the construction and management of their panel until after they have been caught unaware by them.

For instance, one firm conducted a major account satisfaction survey and was shocked to find that not a single customer in the Pacific Northwest had answered its survey. It turned out that no customers from the Pacific Northwest had been invited, as they had all been invited to take a different survey the week before. Obsessive enforcement of invitation limits led to skewed results.

Teams managing house panels often approach the panel in an ad hoc way, rather than with a documented panel-management plan. A good plan can help identify some of the challenges that can be easily addressed with the right approach. Other challenges require making difficult trade-offs, such as whether a panel should be built to be representative or not.

Here are some of the best practices an organization should consider as it seeks to gather the highest quality survey data from a house panel.

Not having a panel

The biggest mistake, for many, is not having a panel in the first place. Many organizations, whether they do one or a hundred customer surveys a year, pull a customer list for each survey. They have their IT, sales or marketing departments create the list from an internal database, sometimes selecting everybody, sometimes selecting customers with a specific profile of buying behavior or demographics; whatever the current survey needs.

These customer lists are not a house panel.

When no one in an organization is responsible for survey data, everybody collects it. Survey tools provide for gathering customer feedback but not for organizing it. Most large organizations today have dozens or even hundreds of separate accounts with survey tools. Employees conduct online surveys to solicit the information they need to answer specific questions. In the process, they degrade the research process. Response rates decline, as too many individuals send out too many survey invitations. Respondents are confused, as questionnaires are published with leading questions and ambiguous lists of choices. It’s easy for do-it-yourself researchers to collect data poorly, leading the business to the wrong conclusions and the wrong decisions.

Panel management is the recognition that customer feedback is an asset and should be treated as such. In order to preserve customers’ willingness to respond to surveys, organizations need to control and conserve access to those customers for survey research. Too often, organizations survey all or most of their customers rather than a random sample, which - at the cost of modestly lower statistical validity - enables many more surveys to run in parallel. Good panel management practices treat survey respondents as an ecosystem and make sure not to hunt respondents to extinction.

Once an organization has built a panel, it needs to work hard to avoid mistakes and follow best practices that will facilitate success.

Align your design

Align your panel design with the types of decisions you will make and the reliability you need. If you need to estimate the percentage of your customers with certain attitudes or propensities to purchase, then every customer should have a random and equal chance of being invited to surveys. If you are more interested in qualitative insights (directional but not projectable), you can take a more ad hoc approach to recruiting panelists and worry less about a structured approach to your panel research.

How hard or easy it is to build a representative panel depends in part on the nature of your business. This is easy for companies with a known and documented customer base, such as e-commerce businesses and B2B firms with large accounts. It is hard for retailers. And it is practically impossible for manufacturers and others who sell indirectly.

If a representative panel of all customers is impossible, consider building a panel of loyal customers instead. For a retailer, rather than creating a panel to represent the average customer, work with your brand’s loyalty program to create a panel of those customers.

If your only relationship with customers is through warranty cards and calls to your customer service center, forget trying to build a representative panel. Use the panel for general insights and for gathering rich qualitative information. The good news for a qualitative panel is that you can be creative about promoting it: you can use corporate Web sites for recruitment, run banner ads across your organization’s sites marketing the panel and include promotional materials for the panel with your online and offline communications to customers.

When your goal is building a representative panel of your customers, you need to zealously guard that representativeness. Don’t consider creating the panel as a “set and forget” exercise. Add new customers to the panel on a regular basis; where possible, create automatic synchronization routines between your customer relationship management (CRM) system and your panel management system. Monitor unsubscribes to make sure one type of customer isn’t leaving in disproportionate numbers, thereby decreasing panel representativity; run periodic profile reports on remaining panelists and compare to your CRM data. Finally, take care when enforcing exclusion rules (where you limit participation based on how recently the panelist was invited to a prior survey). If an earlier survey was to a specific subsegment, then enforcing exclusion rules will keep that subsegment from participating in the current survey, giving you an unrepresentative sample.

Incentive program

Many times when companies are building a new panel they are convinced that they must offer an incentive program. After all, they are creating the panel because of the decline in response that they have witnessed in recent surveys. They hope an incentive system will bribe customers to respond.

The trade-off is that incentive systems can corrupt data quality. An incentive system that is too generous will encourage fraudulent behavior on the part of consumers. We’ve seen rich incentive offers get repeated on coupon and discount blogs (which have become increasingly popular during the Great Recession), leading to an influx of survey participants from outside the panel. As a best practice, send each panelist a unique link to prevent subsequent survey completes. Commercial online access panels are in a constant balancing act of offering incentives rich enough to entice people to join the panel but not so rich as to encourage cheating and fraudulent responses.

While attractive incentive systems are necessary for general-purpose panels, where the panel has no pre-existing relationship with the panelists, financial incentives should not be necessary when researching customers. Customers agree to take surveys out of social norms rather than market norms: they recognize they have a relationship with your firm, however tenuous, and provide you feedback based on that relationship. Accordingly, instead of promising rewards for future behavior, provide awards to recognize past survey-taking behavior. Perhaps your marketing department has leftover items from lapsed campaigns: T-shirts, mugs, pens and so forth. While modest, these things can be used to show that you appreciate respondents’ time.

Try to build a panel without incentives first and see if you can get the response rate back up through better surveys sent to each customer less often. If that does not work, then add an incentive system. It’s hard to switch from an incentive-based panel system to one without incentives, but easy to add incentives. When you first add incentives, focus on small tokens of appreciation rather than major financial rewards to a few winners - it avoids regulatory and compliance headaches that will drive your legal department crazy.

These are real people

Many researchers started doing surveys of the general public, of consumers in general, or of other companies’ customers. With this background, it’s easy to think of respondents in the abstract, as an unlimited resource, and not really worry about how tedious a survey you are subjecting them to.

It’s a mistake to treat surveys of customers the same way. Few firms approach the survey research as another way customers experience the brand, but panel managers certainly should. These are real people, who do business with your firm and they need to be treated as customers. At a minimum, the survey should use a template created by marketing to showcase brand logos, colors and imagery. Even better if the language of the questions themselves reflects the customer experience brand. For instance, Domino’s asks in a follow-up survey, “We want your ordering experience to rock. How was it?”

Besides showcasing the brand, customer surveys should place a higher value on customers’ time than is generally given to respondents’ time. Keep the questionnaire as short as possible. A key way that panel management can assist this is to embed information about the customer into the survey: facts from each panelist’s profile can be used to drive skip patterns and branches. The IT department can integrate CRM with the panel profiles: professionals can then easily target groups of customers for particular surveys and customers themselves see shorter questionnaires, with information they have already provided the organization embedded behind the scenes.

Failing to track the health

Another common mistake is failing to track the health of the panel. The fundamental measure to focus on is tracking the rate at which panelists unsubscribe from the panel. High attrition diminishes the representativeness of the panel and is a sign that you are inviting members to surveys too often and typically to surveys that are tedious.

Another key measure to track over time is the participation rate to surveys. What percent of panelists invited to take a survey actually start the survey? Work to keep that percent steady or even increasing over time. When it starts to decrease, it is an early indicator that you are surveying too often.

Approach the panel as if it were a pond or lake that you didn’t want to overfish. Few managers of house panels centrally coordinate research plans, looking at what surveys will be fielded to the panel on a month-by-month basis. Prepare a calendar of scheduled surveys, with the specific groups that you are targeting. Try to build a calendar that will keep panelists from being invited to more than one or two surveys per month. Invite panelists too frequently and they will stop reading your survey invites.

Easy to make mistakes

Because management of house panels is new to most firms, it can be easy to make mistakes that will compromise data quality. Panel quality is no longer just an issue for commercial vendors to worry about. By following panel-management best practices, however, proprietary panels can help organizations gather customer feedback more rapidly and more economically.