Follow these guidelines to avoid an unsuccessful shopping program

When mystery shopping goes bad

Editor’s note: Peter Gurney is managing director of Kinesis, a Seattle customer experience management firm.

Mystery shopping seems like such a good idea: send people into stores to observe service and report what they see, then use the observations to reward managers, coach employees, track improvements, and compare service levels across the company.

Why, then, are so many managers tearing their hair out in frustration over their companies’ mystery shopping programs? Despite the fact that many businesses spend thousands of dollars annually on such research, an astonishing number of mystery shopping programs fail outright or limp along year after year, perennially under-performing against expectations.

Companies recite a litany of complaints, including:

• disputed findings by employees and managers;
• questioning of mystery shoppers’ skills and credibility;
• more internal administration than planned;
• flat trend lines and undifferentiated scores;
• little or no correlation between mystery shopping results and customer satisfaction ratings;
• lack of timeliness and responsiveness from mystery shopping vendors; and,
• difficulty demonstrating return on investment.

There is nothing inherently faulty about the mystery shopping methodology, which is simply a type of observational research. It can and does provide tremendous value when it is designed and executed well.

So who is responsible for the fact that so many mystery shopping programs go bad?

Let’s examine the suspects:

The industry. Despite the fact that it has been around for at least 60 years, the mystery shopping industry is still immature. It’s highly fragmented with many small regional or specialty firms and relatively few large national players. There are as yet no generally accepted procedures for the selection, training, or certification of mystery shoppers, nor are there clear standards of quality control. As a result, the reliability of the data collected is sometimes questionable. Data quality may vary considerably, not only from vendor to vendor, but also from shopper to shopper within a single study. When employees who have been mystery shopped dispute the results (as they inevitably will, particularly when the data are linked to incentives) it can be difficult to argue that the mystery shoppers know what they are doing.

This situation appears to be changing. A trade group called the Mystery Shopping Providers Association (MSPA) has been formed, with the express goal of “instilling confidence in the business community that mystery shopping is performed in a professional and fair manner.” The MSPA holds conferences twice a year and provides information and resources for mystery shopping vendors and users.

An equally promising development is that independent mystery shoppers have begun to organize and police themselves, using online chat rooms and Web sites as their communication channel. Dozens of local and regional mystery shopping groups have been formed in the past few years. Members of these loose-knit organizations share experiences, discuss techniques and compare the practices of the mystery shopping vendors they work with. Some groups go so far as to rate vendors and even blackball companies that are slow to pay or exhibit questionable ethics. In effect, a grassroots guild of mystery shoppers is developing through the medium of the Internet, creating standards and practices that the industry has heretofore failed to provide.

The vendors. Mystery shopping is a highly labor-intensive activity, and the vendors who supply this service typically possess considerable logistical acumen. Their skills lie in the hiring, scheduling, deployment, and management of a distributed workforce. Unfortunately, they’re typically less sophisticated when it comes to other important factors necessary to the success of mystery shopping programs, such as research design and analysis, development of enterprise-wide service initiatives, integration of data across methodologies, and the linking of performance data to incentives and training. Mystery shopping vendors tend to view their product as a commodity - an attitude reinforced by the vendor selection process used by many companies. As a result, they often function as order-takers rather than partners in the development and execution of strategic quality initiatives.

Mystery shopping suppliers also have an unfortunate tendency to over-promise. This may be perceived as necessary to win accounts, but it results in unrealistic expectations and inadequate planning by the purchasing companies. Vendors sometimes make inflated claims about the rigor of their shopper selection and training, the number of shoppers who work for them, their ability to field shops in a timely manner, and the speed with which they are able to provide data. As a result, vendors frequently get in over their heads, and a relationship of defensiveness and finger-pointing develops between the vendor and the purchasing company. This situation is not likely to change until purchasers become more knowledgeable and realistic about the limitations of mystery shopping and until vendors muster the courage to turn down accounts they know are potentially troublesome.

On the positive side, a few mystery shopping companies are now becoming more sophisticated and rigorous in their approach. Some have designed Web-based programs for hiring, training, certifying, and deploying mystery shoppers, resulting in wider coverage, greater consistency and better quality control. These systems also provide online reporting within hours instead of weeks, making the data considerably more useful to purchasing companies.

Executives. Mystery shopping programs are typically developed in response to broader service quality initiatives. Conceived at the top tiers of the organization, these initiatives may be well-intentioned but they are often vague about deliverables; it is up to the middle level of the organization to work out the details. Lacking a clear roadmap and specific objectives, departmental managers will create programs that serve their own needs but that are not necessarily coordinated with the broader initiative. Thus, people in the operations, marketing, training and human resources departments may all end up running overlapping service quality programs that provide redundant data without being effectively integrated. For example, marketing typically “owns” customer satisfaction, branding and image research, while operations owns mystery shopping, call-monitoring and other performance measures. Seldom are these programs brought together in a coherent architecture that creates useful synergies and provides an optimal return.

In short, company executives have a tendency to launch big service initiatives without thinking them through. Companies may be surprised when their mystery shopping results don’t correlate well with customer satisfaction, increased revenue, or lower costs. But unless these variables are factored into the program design from the beginning, it’s unlikely that strong correlations will result.

Managers. Mystery shopping should make life easier for managers. It should provide reliable, ongoing data for assessing performance, coaching employees, and increasing profitability. Over time the value of mystery shopping should far exceed the effort required to set up and administer the program. However, achieving this level of value requires specialized knowledge that most corporate managers lack.

This information deficit comes at a high cost. Mystery shopping programs frequently suffer from ambiguous objectives, poor design, lack of buy-in from the field and inadequate internal administration. As a result, managers may spend an inordinate amount of time handling internal complaints and wrestling with their mystery shopping vendors. Employees and field supervisors who should be using the data to improve their service skills instead expend their energy exposing shoppers and discrediting the program. Employing a “shoot the messenger” strategy, corporate managers react by churning through mystery shopping suppliers instead of re-evaluating their approach. As a result they repeat the same mistakes year after year with vendor after vendor. Eventually they lower their expectations to the point where a successful program is defined not by its benefits, but by how few headaches it creates.

Such a scenario is common but far from inevitable. Corporate managers can achieve great value and success from mystery shopping by following a few guidelines:

• Define clear objectives. Considering the high price tag that comes with mystery shopping research, it’s incumbent upon company managers to define their goals in specific and measurable terms.
• Keep it simple. In the interest of internal consensus, mystery shopping programs are often designed by committee, which can lead to the program becoming hopelessly complicated and cumbersome. Unrealistic scenarios and long, complex questionnaires are common, creating great frustration for mystery shoppers and program administrators. In such cases the likelihood of shopper exposure is increased and the accuracy of the observations suffers. Simpler designs work better and provide more value.
• Hire a vendor that can be a partner. Large companies often employ an excruciating bidding process that rarely identifies the best vendor for their needs. They issue lengthy RFPs for mystery shopping that are meant to weed out the weakest contenders, but by asking bidders to commit to overly detailed and inappropriate specifications they effectively eliminate more sophisticated companies at the same time. The typical RFP process creates an environment in which mystery shopping vendors over-promise in order to make the first cut, thus setting themselves up for failure if they win the account. In addition, it treats mystery shopping research as a commodity, regarding it as a bulk purchase of data rather than a high-value quality improvement tool. Companies have more success when they research the market carefully and identify the companies that have the knowledge and commitment to help them build a truly valuable program.
• Obtain buy-in from the front-line. When mystery shopping initiatives fail to meet their potential it is often because the people who are accountable for the results - front-line employees, supervisors, store managers, and regional managers - were never properly introduced to the program. As a result there may be internal resistance, creating an unnecessary distraction from the achievement of the company’s service improvement goals. If an incentive tie-in is introduced to the program this situation becomes magnified. To ensure success, employees throughout the organization must be fully informed and bought into the mystery shopping program before it is launched. Pre-launch efforts should include training on how to read mystery shopping reports, how to use the information effectively, and how to set goals for improvement.
• Provide adequate internal administration. Few companies anticipate the amount of administration necessary to run a successful mystery shopping program. A common mistake is to assign young, inexperienced personnel to handle the program - a sure prescription for chaos. It requires a strong administrator to keep the company focused and on board, and to make sure that recalcitrant field managers are not able to undermine the program before it stabilizes and begins to realize its potential value.
• Plan for change. Even well-designed and administered mystery shopping research requires periodic adjustment. Performance scores eventually flatten out or cluster together, diminishing the value of the program as a tool for rewarding top performers and continuously improving quality. Periodic reviews should be worked into the program design so it can be kept relevant and useful, and so the bar can be repeatedly raised on service quality and employee performance.

Meeting the demand

The consumer demand for better service is growing all the time. Companies struggle to meet this demand in the face of high employee turnover, shrinking profit margins, and increasing competition. At the same time the business landscape is becoming more and more complex, with 24-hour, multi-channel service now a basic consumer expectation.

Mystery shopping is among the more powerful tools available to companies seeking to improve their service quality. Providing objective data about service execution across locations and delivery channels allows managers to identify specific areas for improvement and to reward employees in a consistent, relevant manner. Despite the problems many companies have experienced with mystery shopping, the need for this research will only increase as customer demand for high-quality service grows.

Improving the way mystery shopping is conducted will benefit consumers, companies, and suppliers. Changes in the mystery shopping industry and in the capabilities of some vendors are leading to better data collection practices, but much progress remains to be made. On the other side of the equation, if companies are to benefit from these changes they need to improve the process by which they initiate, design, and administer mystery shopping programs.