Editor's note: Jim Bryson is president of 20/20 Research, Inc., Nashville.
Mystery shopping, when used properly, is a very useful tool that is sometimes misused in a company's market research portfolio. It has long been a research staple for many sales and service organizations, who use it to check operational procedures by using a "shopper" to see that the employee smiles, says the right phrase or that the plant in the corner is getting enough water. However, sometimes marketers make the leap of faith that adherence to measurable operational policies leads to improved customer service and satisfaction.
In practice, mystery shopping is a holdover from companies that are operations driven and often misses the mark for companies that wish to be customer driven. Much has been said in the last few years about customer satisfaction and the fact that customer satisfaction is the wave of the future. Everyone has heard that service is the key to keeping customers and that new customers are "X" times more expensive to obtain than current customers are to keep. However, one should remember that customer satisfaction is not a laundry list of operational procedures but a perception formed in the customer's mind of the service received.
Therefore, mystery shopping is not a customer satisfaction measure but an operational measure which may or may not have anything to do with true customer satisfaction.
The proper use of mystery shopping is to measure operational procedures and should only be used with customer satisfaction measures when there is a confirmed, direct correlation between the two.
Domino's Pizza has an excellent mystery shopping program. Domino's has determined that 30 minute pizza delivery is directly correlated to customer satisfaction. Thirty minute pizza delivery happens to be an easily measured operational variable which is well suited for a mystery shopping program. Therefore, they recruit ordinary consumers in each of their store areas to order a pizza once a month and report whether the delivery was made within the 30 minute standard. They are measuring an operational variable which they know is directly correlated to customer satisfaction and around which they have built their entire marketing campaign.
There are advantages if mystery shopping is used properly; however, it also has potential pitfalls which should be carefully avoided.
1. As illustrated in the Domino's example, mystery shopping is excellent for measuring operational procedures. The potential pitfall is that operational procedures are easily measured and therefore they are sometimes used as surrogates for customer satisfaction. For example, a convenience store chain requires all their employees to say "Have a good day" when the transaction is completed. The concept is great, but the employees usually act more like they are being tortured than really caring about your happiness.
2. Mystery shopping is also good for checking that decor, signage, employee dress, etc. conforms to company standards. The potential pitfall is that these variables are not easily changed and any company employee can inspect these on a surprise visit. Outside research firms can be a waste of time and money to measure these types of variables.
3. One of the greatest advantages to mystery shopping is that the variables are usually very easy to measure. Therefore, goals and policy are easy to set and measure. The potential pitfall is that managers are often tempted to use these easily measured variables exclusive of other research tools to measure their performance and progress.
There are several disadvantages to a mystery shopping program.
1. The program is relatively expensive if there are many locations to be shopped. A typical cost is between $40 and $60 per mystery shop before other elements of the program are included.
2. Most mystery shopping programs are objective in nature and miss the subjective "feel" of a location. There is no measure of the atmosphere of the location or the unspoken attitude of the employees even though they are often key to the overall satisfaction of the customer.
3. The mystery shops themselves are usually performed by professional interviewers or others who complete many shops in a short period of time. Although many mystery shopping programs are beginning to incorporate subjective measures, professional interviewers often become biased in their judgments. They become either numbed or overly sensitive to the atmosphere and attitude issues mentioned previously. Therefore, even attempts at measuring subjective variables in a mystery shopping program have questionable results.
4. Mystery shopping does not measure the most important variable of all- the customer's perception of the service. After all, it is the customer's perception that really counts, not whether an employee diligently followed each and every operational procedure in the handbook.
One common misuse of mystery shopping is to use it as a measure of customer satisfaction. This point has been made thus far in this article. Another common misuse that needs to be addressed is the practice of using mystery shopping results in the evaluation of managers. Since mystery shopping is relatively expensive, a manager's evaluation usually consists of very few mystery shops. This is inherently unfair to the manager since two or three mystery shops may not be indicative of the normal state of the manager's responsibilities.
An alternative to a mystery shopping program is to gain customer feedback on their experience as quickly as possible with a technique known as post transaction sampling. This methodology focuses on the customer and his/her perceptions rather than the employee and his/her adherence to policy. Often post transaction sampling is used in tandem with mystery shopping to harvest the best of both methodologies.
The advantages to post transaction sampling versus a mystery shopping program are:
1. Post transaction sampling emphasizes the customer's experience and perceptions rather than a paid interviewer's.
2. As a rule, it is more subjective in nature. Customers will not be able to give detailed information about the operations but will be able to replay their perceptions about atmosphere and employees' attitudes.
3. The customer being interviewed about their experience develops the impression that the client really does care about their business since they are taking time to call and check on them.
4. The cost per interview is generally 25%-40% of the cost of mystery shopping. This means that for the same amount of budget, two and a half to four times as many interviews can be completed yielding a much more reliable picture of the performance at each location.
5. Performance and changes over time at individual locations are monitored better since the sample sizes are larger allowing more confidence in the results.
This methodology has two major disadvantages:
1. Operational procedures determined to be directly correlated to customer satisfaction are not measured. The more time between the transaction and the interview, the less objective information the customer will be able to remember. Long term perceptions will still be valid; operational variables may not be available. This is one reason some type of mystery shopping program is often conducted along with post transaction sampling.
2. Post transaction sampling can be an administrative nightmare. To be effective, the sampling should take place within two to three days of the transaction. Many companies have difficulty quickly producing names of customers who have conducted transactions.
A mystery shopping program should be an integral part of the customer satisfaction program when the performance of a particular service can be identified as contributing directly to improved customer satisfaction (e.g., Domino's Pizza). It should not be used as a surrogate measure of customer satisfaction simply because it is easy to set up and measure.
Also, unless the research budget is much bigger than most, mystery shopping can be an unfair means of evaluating a store's performance.
We have found it worth the effort to combine post transaction sampling with mystery shopping to form an effective and comprehensive measure of both the customer's perception of service and the employees' adherence to operational policy.