How to address survey manipulation by auto dealerships

By David Ensing | October 1, 2008

Reading time: 11 minutes

How To Address Survey Manipulation By Auto Dealerships

Abstract

While auto dealers' attempts to influence the customer satisfaction survey process can range from benign to something worse, the impact of their efforts is the same: the corruption of the methodology. The author, whose firm conducts a number of auto dealer satisfaction programs, offers strategies that address the problem and outlines ways to work with the dealerships to implement acceptable practices.

Research Topics:: Customer Satisfaction Studies | Respondent Cooperation/Satisfaction
Industry/Market Focus:: Automotive | Consumers
Content Type: Research Methodology | Magazine Article

Share Print

Let me help you with that

Editor’s note: David Ensing is director, research and development, with Maritz Research’s Automotive Research Group, Maumee, Ohio. He can be reached at 419-725-4055 or at dave.ensing@maritz.com.

Concerns about auto dealership attempts to interfere with the customer satisfaction measurement process have been around for a long time, but lately they seem to be intensifying. Clients are asking how to guard against survey manipulation, there have been articles and letters to the editor on the topic in Automotive News, and new methodologies employed in customer satisfaction research have led to new opportunities for survey manipulation. As a provider of automotive customer satisfaction research, Maritz Research offers this article as a perspective and thought-starter on defining and addressing survey manipulation.

Difficult to determine

The exact frequency of survey manipulation within the automotive industry is difficult to determine because it depends on how survey manipulation is defined. An August 2007 Automotive News article reported that nearly 50 percent of respondents to a TrueDelta poll of about 1,700 new car buyers said the dealership tried to influence their survey responses. However, upon examination of the full report available on TrueDelta’s Web site, it is apparent that over half of the survey influence attempts reported were, “they asked me to let them address any problems rather than reporting them.” Some would define this as survey manipulation; others would define it as good customer service. Specifically, customers responded in the following ways to the question, “When you purchased your car, what did the salesperson or manager say about the survey?”:

25 percent - They never mentioned the survey.
30 percent - They mentioned that I would receive a survey (but none of the following).
36 percent - They asked me to let them address the problems rather than reporting them.
28 percent - They asked me to give them perfect scores but did not beg.
8 percent - They begged me for perfect scores.
9 percent - They said they would only get a bonus if I gave them perfect scores.
2 percent - They asked me to bring the survey in to the dealer and fill it out while they watched.
2 percent - They asked me to bring the blank survey to the dealer so they could fill it out.
2 percent - They offered me a gift in exchange for one of the above.

As noted by TrueDelta, in the approximately 45 percent of cases where dealership attempts to influence the survey were reported, tactics were relatively mild (asking for perfect scores or asking for a chance to correct any problems) and “in contrast, the relatively extreme forms of corruption were much less common than anecdotal evidence suggests.”

Several important conclusions can be drawn from this study. First, most dealerships do not attempt to influence survey results, at least not by making requests to customers. Second, in many cases attempts to influence responses may actually be appropriate and beneficial to customers (offering to fix any issues before they are reported). Finally, a small minority of dealerships engage in blatant forms of survey influence, but they are contributing to a poor reputation for all dealerships.

Exist on a continuum

As can be seen from the list above, dealership efforts to influence customer satisfaction survey responses exist on a continuum. Some efforts can be seen as good customer service and may be encouraged, or at least accepted, by manufacturers. Other efforts are clearly dishonest and are only directed at increasing dealerships’ customer satisfaction scores. Unfortunately, many practices fall somewhere in between. Part of the difficulty of defining survey manipulation is determining where on this continuum a dealership crosses the line from acceptable practices to unacceptable survey manipulation.

Every manufacturer (preferably with input from its dealers) will need to explicitly define what it considers acceptable and unacceptable practices. Also, many instances will need to be determined on a case-by-case basis because all forms of possible survey manipulation cannot be anticipated. As a general rule, at Maritz we define survey manipulation as anything dealership personnel do to encourage customers to misrepresent their actual dealership experiences or to systematically prevent the survey process from accurately measuring the overall dealership experience (e.g., encouraging satisfied customers to respond to the survey while not offering the same encouragement to less-satisfied customers).

What to do about it

Once manufacturers have decided what constitutes survey manipulation, they have to determine what to do about it. The first step most manufacturers and suppliers take is to put processes in place to identify dealerships that attempt to manipulate survey results. While this step is extremely important, we believe it only addresses part of the problem. To effectively address this issue, manufacturers, dealers and customer satisfaction measurement suppliers need to work together and focus on:

Establishing, communicating and consistently enforcing a strong policy prohibiting survey manipulation.
Designing survey systems that make manipulation of survey results difficult.
Setting up systems to identify dealerships that attempt to manipulate survey results.
Working with dealerships to implement acceptable practices.
Designing reward and compensation programs that minimize dealerships’ motivation to manipulate survey results.

Our advice on each of these issues follows.

Establishing, communicating and consistently enforcing a strong policy prohibiting survey manipulation. This is probably the most important and least-executed step required to control survey manipulation. Engaging in survey manipulation is a risk/reward decision. If there is very little risk because there are no or few consequences of getting caught, some dealerships will engage in the practice regardless of how many processes are put in place to minimize its occurrence. Consequences of engaging in unacceptable practices need to be specified in advance and communicated to dealerships and these consequences need to be enforced if dealerships are identified as engaging in survey manipulation.

A good example of a company that strictly enforces a policy prohibiting survey manipulation is Enterprise Rent-A-Car. At Enterprise, an important criterion for employee career advancement is the branch ESQi (i.e., customer satisfaction) score. Given the importance of this score to employees, one would think survey manipulation would be rampant. It is not, however, because Enterprise’s executive management clearly communicates the importance of this measure and there is a company-wide understanding that manipulation will not be tolerated.

Being that dealerships are independent businesses, automotive manufacturers do not have direct control over their dealerships and dealership personnel as Enterprise has over its branches and branch personnel. Therefore, manufacturers will need to utilize consequences which they have the ability to enforce. Effective consequences we have seen include:

Removing manipulated surveys from the scoring process.
Requiring dealerships to reimburse the manufacturer for the cost of conducting their CSI surveys for the time period under which manipulation occurred.
Denying customer satisfaction-based compensation or rewards to dealerships that have manipulated their scores.
Subjecting dealerships that have been identified as manipulating their scores to audits of all or many of its manufacturer programs.
Including language in the manufacturer/dealer franchise agreement that customer satisfaction survey manipulation is grounds for removal of the franchise.

Designing survey systems that make manipulation of survey results difficult. Manufacturers and their suppliers should design systems that make survey manipulation as difficult as possible. These systems need to be continually monitored and updated to address new methods of survey manipulation. Some ways to make survey manipulation difficult include:

Use contact information that can be verified. A simple way dealerships can manipulate the survey system is to provide false customer contact information. By doing so, the dealership personnel ensure that unsatisfied customers are unreachable or they can even intercept the contact and respond to the survey themselves. Therefore, when possible, it is advisable to use contact information that can be verified.
Don’t conduct point-of-purchase/point-of-service surveys. While this is probably one of the easiest ways to contact the customer, it is easily intercepted by dealership personnel and it is practically impossible to verify that the customer is truly responding to the survey.
Use contact information that can reach all, or almost all, customers. Having the ability to reach all customers is important for sampling reasons, but an added benefit is that it helps reduce survey manipulation. If contact information that covers only a percentage of customers is used (e.g., e-mail addresses) it is much more difficult to identify instances where dealership personnel omit this information when a customer is upset.
Use multiple contact methodologies. Using a multiple contact methodology (first attempting to contact a customer by e-mail, then moving to mail, phone or both) makes it more difficult to influence the survey process because multiple sets of contact information would need to be manipulated.
Allow customers the option of remaining anonymous. If customers know that dealership personnel will not be able to tie their responses to their names they will be less likely to give in to pressure from dealership personnel to provide them with good scores.

Setting up systems to identify dealerships that attempt to manipulate survey results. When most of us are on the highway and we know there are no police to monitor our driving, we tend to speed more than we should. Similarly, the less likely dealerships are to get caught manipulating survey results, the more tempting it will be for some of them to try. Therefore, processes need to be put in place to identify potential survey manipulation. These processes will be specific to the given project and the methodologies used, but can be broadly categorized as follows:

Examine customer contact records. One of the most frequently used methods of survey interference is for dealership personnel to intercept surveys by providing false customer contact information. Customer contact information should be examined for duplicate or fake addresses, phone numbers, e-mail addresses, etc.
Examine incoming materials. Incoming surveys can often indicate that manipulation has occurred. This ranges from fairly obvious cases where several paper surveys from different customers are included in the same return envelope, to the less obvious (but identifiable) cases where Internet-based surveys show signs of manipulation.
Examine customer comments. Examining customer comments is one of the easiest ways to identify survey manipulation and one of the few ways to identify when customers have been pressured to give high ratings. Often, when customers are pressured to give good ratings, they say so on the survey.
Examine contact resolution reports. Calculate contact resolution at the dealership level and identify dealerships that have unusually high levels of invalid contact information. These dealerships may be entering false contact information when customers are upset, thus making these customers unreachable.
Examine the actual data. Dealerships that engage in survey manipulation might show distinctive data patterns in their survey results such as highly variable data, sudden changes in data or increasing response rates nearing the end of a scoring period.
Randomly audit survey responses. The best way to determine if customers have been pressured to misrepresent their experiences at dealerships is to ask them. Unfortunately, many dealerships balk at this process because they feel that by doing so the manufacturer is instilling distrust between the customer and the dealership. A potential compromise is to conduct random audits of survey respondents where a relatively small number of customers are telephoned and asked about survey manipulation efforts.
Put a survey manipulation question into the survey. This practice can be helpful and it has been used in several customer satisfaction surveys. It helps identify when customers are being pressured to misrepresent their dealership experiences but it does not address survey manipulation in the form of dealer personnel intercepting surveys or providing invalid contact data.

Working with dealerships to implement acceptable practices. There are many practices dealerships can implement that both increase customer satisfaction scores and improve the customer experience. If manufacturers embrace these practices and work with dealerships to implement them, dealerships will have less need to engage in unacceptable survey manipulation. Some practices we have seen encouraged by manufacturers include:

Showing the survey to all customers and requesting that they return it.
Asking customers if they are unsatisfied about anything regarding their experience and attempting to resolve the issue.
Telephoning customers within a few days of a sales or service event, inquiring about their satisfaction and engaging in appropriate efforts to resolve any dissatisfaction.
Explaining the importance of receiving customer feedback for both the dealership and the manufacturer.

Designing reward and compensation programs that minimize dealerships’ motivation to manipulate survey results. Finally, we need to examine why dealerships are motivated to manipulate survey results. The answer is simple: In many cases the difference in a few customer satisfaction surveys can mean a lot of money for the dealership. While it is important to hold dealerships accountable for their treatment of customers, and compensation based on customer satisfaction scores is the most obvious way to do that, we believe the way in which some reward and compensation programs have been designed has exacerbated the problem of dealerships attempting to manipulate the system. Some suggestions for setting up programs that minimize the motivation to manipulate results while still holding dealerships accountable for customer satisfaction are:

Compensate/reward based on several desired business metrics of which customer satisfaction is only one. Use of additional desired business metrics has several advantages. First, customer satisfaction scores are less likely to be manipulated because their weight in determining compensation is reduced. Second, and perhaps more importantly, dealerships can focus too much on customer satisfaction. We have heard of cases where dealerships have refused to sell vehicles to customers because they were afraid the customers would give them poor customer satisfaction scores. Obviously, this is to the detriment of both the dealership and the manufacturer. If number of sales was also a criterion in the compensation program, this would be much less likely to happen.
Use a tiered compensation strategy rather than an all-or-nothing strategy. Dealerships will be less likely to manipulate survey results if the stakes of missing a target score are lessened. A tiered program with graduated rewards is both more motivating for dealerships (especially those that have little chance of reaching the highest level) and reduces motivation to manipulate the survey process because less differential compensation is at stake between reward categories.
Separate the dealership performance appraisal process from the dealership diagnostic process. When dealerships try to affect performance appraisal scores (so they get compensation from the manufacturer) they also contaminate their diagnostic results. Therefore, they don’t know what processes actually need to be improved at the dealership. One way to address this issue is to conduct performance appraisal evaluations separately from diagnostic evaluations. While dealerships may still attempt to manipulate their performance appraisal evaluations, manipulation of their diagnostic information will be to their detriment.
Require large sample sizes for determining scores. Obviously, the smaller the number of responses that make up a dealership’s score, the greater weight each response has in determining the score. Therefore, changing one or a couple of responses can have a large impact on the final score. If manufacturers limit their reward and compensation programs to annual or semi-annual awards, fewer end-of-term manipulation efforts will take place because it will be more difficult to move the overall score. Manufacturers might also consider different award periods depending on size of dealerships.

A systemic issue

We believe survey manipulation is a systemic issue, but it affects vehicle manufacturers differently. Each manufacturer, with input from its dealerships, needs to decide what practices are acceptable and unacceptable. Each manufacturer needs to determine how seriously it wants to pursue this issue and how serious the consequences should be to its dealerships who engage in unacceptable practices.