Worth the effort

Editor’s note: Jarad Carleton is an IT consulting analyst at Palo Alto, Calif., consulting firm Frost & Sullivan. This article is excerpted from a Frost & Sullivan white paper provided by Seattle research firm Global Market Insite Inc.

Regardless of industry, regional economy or geopolitical boundaries, organizations around the world will always research and collect data on markets of interest. To make the research data relevant for the researching organization, panels are often created to provide specific targeted data on key subsegments of a given population. Some organizations develop panels by sending people into the field to conduct targeted surveys, others send out surveys via traditional mail, and others survey target populations via the telephone.

These methods of panel-based market research have been in use for decades, but societal changes have decreased the efficacy of mail and telephone research and other methods used in conjunction with panel research such as mall intercepts have become prohibitively expensive. As the old traditional methods of panel research become less effective and more expensive, online research tools have been created to satisfy the need for rapid, accurate and highly targeted market research.

Despite the fact that online panel research does offer quantifiable time and cost savings, questions remain concerning the validity of online research panels. Questions pertaining to representative samples, duplication and motivation of panelists, and ensuring that panel members are who they claim to be are issues that online panel providers must adequately address to establish long-term credibility with the organizations they seek as customers.

Limited resources

Every organization regardless of size has limited resources to commit to panel research. Although traditional panels provide important market information, methods such as telephone panels or mall intercepts can be prohibitively expensive sources of market information due to labor and travel costs. In the case of telephone panels, depending upon the size and geographic focus of the study, an organization typically pays five to 100 operators to make calls in a target market that can only be completed in a three-hour frame from 6 to 9 p.m. if the research is being conducted in a single time zone.

Consequently, for the study to collect data in a timely manner, it is in the best interest of the organization conduct research in multiple time zones and hire as many operators as possible at a cost ranging from $15 to $20 dollars per completed interview. Complicating matters further, in order to have a true telephone panel, a qualified list of volunteer panel members is required such that operators call specific people rather than dial randomly from a pre-assigned list of numbers. Since qualified lists are rarely available, companies tend to purchase lists from magazine publishers and other sources at a cost of 5 cents per name. The problem is that a list of 10,000 names may only lead to 100-200 completed interviews with the desired target population.

In addition, each person called must be screened for desired traits such as ethnicity, income level, education level and other factors that will lengthen the duration of the call and increase the cost of the study. Supplementing the data obtained from a phone or mail panel with mall intercept research methods can lead to a more targeted study, but costs quickly add up with this technique as well. Expenses such as labor, travel, remote office space and data entry can push costs above $10 per completed interview, which can be quite expensive if 1,000 interviews are needed for statistical significance.

In almost all organizations, budgetary issues will arise when panel research is an integral part of a market research study. As a result, more organizations have begun actively utilizing outsourcers that specialize in online panels.

Keeping them engaged

One of the more difficult challenges for organizations that conduct primary research on an infrequent basis lies in the management of online panels and keeping panel members engaged through personalized communities. Most organizations are poorly informed regarding the difficulty of these tasks and mistakenly assume that a Web site and Access database are all that is needed to manage a panel.

That assumption couldn’t be further from the truth as there is significantly more to online panel management than a simple database and static Web site. There is a considerable amount of work and communication involved with panel management including, but not limited to:

  • obtaining a representative mix of people on the panel;
  • ensuring that e-mails make it past spam filters and to panel members;
  • keeping track of new e-mail addresses;
  • checking to ensure that panelists are who they claim to be;
  • tracking responses from panel members to ensure questions are answered properly;
  • tracking and paying incentives to millions of panelists; and
  • answering panel member inquiries in potentially several languages.

There is also the challenge developing a large enough panel such that members don’t become exhausted from endless survey requests and quit the panel. Conversely it is also important to avoid underinterviewing panel members, which creates another set of problems including panelists that don’t remember volunteering to take surveys, old e-mail addresses, no response to survey requests, and requests to opt-out of the panel.

Attrition of panelists is a tricky problem to solve when the level of contact isn’t carefully managed. Unfortunately, proper panel contact management is an expensive full-time project for medium and even large enterprises with resources to devote to the task. Because most companies don’t understand the complexities of panel management, it is common for companies to underestimate management costs by 50 to 75 percent as they fail to accurately project the total expenses involved with panel development and management.

In fact, the costs involved to recruit an active panel large enough to support regular research are staggering (to support regular research, a panel of approximately 100,000 persons is required) and typically range between $250,000 and $300,000 before research even begins (based on average acquisition cost of $2.50 to $3 per person). On top of those costs are additional expenses related to keeping panelists engaged through regular contact or personalized communities, the cost of the technology platform to support the communities, individual profiles, the Web site as a whole, and incentive management. Furthermore, after the panel has been created, panel attrition begins making it necessary to continually recruit additional panelists (again, at a cost of $2.50 to $3 per person).

Time constraints

One of the most common problems facing organizations that conduct panel-based market research is that of time constraints. If it takes too long to collect data, the results become skewed with time and informed business decisions cannot be made based on the research. An issue that arises with panel-based research via the telephone is that there is only a three-to-four-hour block of time each day during which an organization can conduct research. Since most studies have a short four-to-seven-day timeline, more resources must be shifted to call center operations which can rapidly consume more research study funds than what may have been budgeted for.

Another issue that interferes with timelines for panel-based research via telephone is that response rates are falling each year as a result of Caller ID and other call-screening technology. Furthermore, studies that require respondents to answer at specific times of the day or that require all of the data to be collected within 48 hours have become increasingly difficult to conduct. Unfortunately, market research requests have become so common that many potential panelists feel inconvenienced or disinterested, leaving organizations with a smaller pool of panelists to choose from. As market research requests continue to saturate the global marketplace, fewer people are willing to make the time to answer marketing questions in the mall or via telephone when offered little or no incentive to participate.

Difficult to manage

As previously mentioned, internal panel development can be costly and difficult to manage even for large multinational firms that have sufficient resources to undertake the demanding task of maintaining a large panel. And in the event that an organization wants to conduct research in multiple countries, it will have to find employees with appropriate language skills and knowledge of privacy laws different countries, which can make panel-based market research even more difficult.

Outsourcing panel development and management to a third party enables the organization to focus on its core competencies rather than on the difficult task of vetting potential panelists and maintaining the long-term interest of the panel through incentives and community building. It also enables the organization to conduct research simultaneously in different countries without needing to find employees with pertinent language skills and without needing to understand the details of privacy laws in different parts of the world.

Although large multinationals have the ability to navigate issues related to privacy, incentives and community-building, a panel outsourcing company has economies of scale that are difficult to match in areas of quality and price even for a large consumer products organization.

Conducting research via an online panel does require the researching organization to be familiar with issues that lead to panel integrity problems. When a company develops its own online panel or uses one from an unsophisticated market research outsourcer, panel integrity problems can skew the research results.

Examples of panel integrity problems include, but are not limited to:

  • respondents who participate in multiple panels;
  • respondents who sign up multiple times for the same panel using different e-mail addresses and different mailing addresses;
  • respondents who identify themselves as someone they are not, i.e., an executive, of a different ethnicity, a male answering for a female, a 25-year-old female posing as 40-year-old male, etc.

Unfortunately it is impossible to ensure the integrity of all panel participants, but understanding the issues that create panel integrity problems is the critical first step towards ensuring research results are as accurate as possible.

Even more important is the second step, which is having a clear understanding of safeguards that should be put into place to minimize panel integrity problems and make them statistically insignificant.

Safeguarding against panel integrity problems can be a complex task and always requires a multifaceted approach. Although there is little that can be done to prevent a panel participant from enrolling for a panel with another company, it is possible to minimize one person participating in multiple panels for a single company. This is possible by creating a system of checks and balances in which it becomes difficult to sign up multiple times and to participate more than once in the same survey.

One of the most common methods of subterfuge attempted on the Internet today is when a person signs up for multiple accounts using multiple e-mail addresses. This is not difficult due to the fact that a standard DSL account typically gives a person five e-mail accounts before it becomes necessary to start using accounts with free online services such as Yahoo!, Hotmail or Gmail. This strategy should be countered in five ways:

1. Place limits on the number of surveys that can be completed from a single IP address. This is a highly effective measure for static IP addresses. It is also a useful countermeasure for consumer DSL accounts that have dynamic IP addresses due to the fact that the user will have to log off the DSL line and reinitiate a connection before starting another survey. For panelists who are tech-savvy and understand the problem is with the IP address, the inconvenience discourages them from completing multiple surveys. For panelists who don’t understand the basics of Internet technology, this will be a sufficient barrier to prevent taking a survey multiple times.

2. Asking for a mailing address to which earned rewards can be mailed after taking a number of surveys. Some unscrupulous individuals with multiple e-mail addresses and dynamic IP addresses will try to take as many surveys as possible and then try to change all accounts to the same mailing address in order to collect their rewards. When this happens, it is easy to catch and the panelist as well as all of his or her e-mail addresses and mailing addresses can be blacklisted.

3. Time-stamping each response to identify cheaters.

4. Analyze the open-end responses to make sure coherent, thoughtful responses were given.

5. Re-survey 105 of the respondents a week later to see if there is continuity in their answers.

Finally, a program should be put into place that collects basic demographic information from each panelist and compares that information to questions in the survey to ensure that the panelist is answering in accordance with who he or she claims to be. One commonly acceptable practice is to have a panelist begin taking a survey without telling them what it is about. The first questions will be basic demographic profiling questions that should fit with the profile they signed up with. If at any time the answers don’t correspond with their original demographic profile, the survey ends and the company can review the panelist account to determine if it should be culled.

Reduce time and expense

Online panels enable an organization to significantly reduce the time and expense of research projects by eliminating people in the field, travel costs, and the wait for business-critical study results. The result of this fundamental change in panel research has also changed the manner in which companies compete. By reducing the time and cost needed to compile panel research data organizations have been able to learn about the markets in which they participate with greater depth and frequency. As a result, organizations now have the ability to respond faster to changes in the market, which is benefiting enterprise and consumers alike.

Figure 1 shows one example of time saved for research that utilizes an online panel for a small study. Many market research studies do not require a nationally representative sample but instead require a sample of people who use a product or service. As a result, most studies want a broad demographic representation such as age, gender or geography of respondent or select group of users and potential users. For those studies that do require a representative sample of respondents, a representative audience implies that the study was able to obtain information from a pool of panelists that closely resemble the demographic composition of the most recent official census along such characteristics as age, gender, or ethnicity, per geographic region examined. This is accomplished by starting with a large and demographically diverse panel of potential respondents, then sending a pro rata percentage of invitations to each demographic group for which representation is required. The percentage of invitations should match the percentage of this group’s representation in the national census.

The last safeguard that ensures proper representation requires demographic quota cells to be defined within the survey so that only the desired number of respondents in each quota cell or demographic sub-group is allowed to complete the survey. All other respondents are locked out. Ensuring that an online panel produces a set of representative responses is a very difficult goal to achieve with a panel that is managed in-house due to the fact that these panels are usually too small to adequately represent an entire population.

When the panel is large enough to obtain a statistically relevant sample, it is very important to establish how sample was taken. Far too often organizations will use a flawed methodology and send out thousands of invites to take a survey without imposing controls to ensure that completed surveys are in fact a representative sample. Blindly sending out thousands of requests to take a survey will never provide a representative sample because it is impossible to impose quota controls and ensure that target demographics are represented in the raw data.

Furthermore, utilizing online panels increases the complexity of obtaining a representative sample. This is due to the fact that approximately 15 percent of the population in developed countries isn’t online and because young women in Canada, the U.K. and the U.S. are power users of the Internet and many have an affinity towards taking surveys for financial incentives. Therefore to obtain a representative sample, it becomes necessary to spend much more time and expense to manage the panel and invest in expensive and sophisticated software tools that allow the organization to impose and enforce sample quotas. Most organizations cannot afford the investment required to collect representative samples, which is why many companies increasingly rely on panel outsourcing companies that are better equipped to manage and maintain the panels and software necessary for the task.

Embraced panels

Although traditional offline research is still going strong, it is clear that businesses with market research needs have embraced online panel research. For businesses competing in an increasingly global economy, one critical factor is the ability to rapidly obtain accurate information on target markets of interest. Without timely and accurate market intelligence, business decisions pertaining to the direction of a product, service or brand become a guessing exercise with no better than a 50 percent chance of success. Online panel research eliminates a lot of guessing about a market and does so faster and at less cost than traditional offline panel research methods.