Driven to satisfy

Editor's note: William Bailey, Ph.D., is a statistical consultant and market analyst based in Orlando, Fla.

Serving 1.1 million customers in metro New York, Long Island, and seven upstate counties, the Auto Club of New York (ACNY) is the eighth largest affiliate of the Automobile Association of America. In early 1993, the ACNY engaged the author to seek the opinion of its membership on the organization's service quality and provide an evaluation to management.

"Being a membership organization we wanted to make sure that we stayed member driven," says Marshall Doney, ACNY's director, marketing and service quality.

After preliminary discussions, a mail survey was chosen as the best way to approach members. "In the past we received a good response rate from mail surveys and I like the methodology," Doney says. A six-page survey was mailed to 2,500 members. To encourage response, a new one dollar bill was included with a personalized cover letter explaining the study's intent and how the results would be an integral part of the club's long range strategic plan. The letter was signed by Doney.

The questionnaire's central focus was to rate member opinion on 16 service attributes from three perspectives:

  • the importance of each attribute;
  • the level of expectation for each attribute, and
  •  the member's rating of the club's performance on attribute delivery.

Figure 1 illustrates the type of attributes identified as important by members in an initial set of focus groups, and considered in this research phase.

As with most attitudinal surveys, this one sought member opinion toward the organization using basic questions involving services used, favored methods for service usage, competitive comparisons, and overall satisfaction indices.

The auto club also wanted to highlight areas where service delivery (based on performance) fell below expectations for those attributes members said were most important.

Measurable and actionable

ACNY sought to measure service quality as members defined it, as opposed to traditional quality control measures, which are often internal measurements not directly linked to the customer.

"In the past we may have assumed that we knew what members wanted," Doney says. "This research resulted in information that was measurable and actionable. Measurability is critical because we've all done research that gives us a lot of 'nice to know' information but it's not readily actionable by management. When you know what's important and what members expect it makes it easier to allocate resources."

After a series of introductory questions, the respondent was asked to evaluate each attribute and rate its level of importance. The 10-point Likert scale ranged from "not at all important" to "extremely important." Next, the expectation of each attribute was rated. That is, did the respondent expect very little, or were there "extremely high" expectations for the club's delivery of the particular attribute?

For example, there might be low expectations for office location since many services can be provided over the telephone. Thus, having branch offices might also be less important than other services. However, if an office is needed, one might expect it to be clean and professional looking.

Each attribute had a description to narrow the respondent's interpretation, and the scale allowed for registering "no opinion."

The study's results were based on 1,241 member responses, a response rate just under 50 percent. SPSS was used for multivariate analysis, and Lotus 1-2-3 and Harvard Graphics for tabulation, graphic presentation and/or perceptual mapping. Final reporting used WordPerfect for integration of text and data.

Beyond basic descriptive statistics, regression analysis was used to highlight significance differences, and the attributes were classified using factorial analytic techniques based on how members responded to the question on the club's performance.

Using this analysis, two distinct factors were detected and described as "service access" and "service delivery." (For the analytical reader: The classification is considered excellent based on the Kaiser-Meyer-Olkin statistic [.93] that tests partial correlations. Further, reliability tests were performed on each factor. The Reliability Alpha, which tests the internal consistency of variables in each factor, ranged from +.81 to +.91 and is considered excellent.)

To maintain simplicity, data presentation included bar and pie charts, and detail tables. However, the actual interpretation was based on the comparison between an attribute's importance mean rating and the difference between the attribute's mean performance and its mean expectation, called the P-E gap. (Refer to Figure 2.)

A positive P-E gap for an attribute is favorable and means that the organization's performance exceeds expectations. This study found most of the gaps to be negative; that is, performance is below expectations. However, one can consider a new line of interpretation when you allow for attribute importance. Further, the interaction between the mean response for importance and mean P-E gap highlights the very significant attributes.

ACNY management found the perceptual map shown in Figure 3 very useful. The map used in this research does not form its basis on pure quadrant theory but is an extension of an earlier theory that involved "market acceptance" models using "cash cows" and "dogs" to segment products based on their market attractiveness and competitive positioning. Here we relate attractiveness to importance, while the P-E gap is a gauge of perceived market position. Figure 3 clearly exposes attributes that have a high level of importance to the membership, as defined by its mean rating, but have a low performance-to-expectation deviate (P-E gap).

When the P-E gap is considered, those attributes with a large negative value (performance below expectation), warrant further investigation. Within the secondary box displayed in Figure 3, there clearly is one attribute that needs immediate management attention. There also are several that are simply "average" when compared to the overall rating means (that define the quadrants). For the organization to improve, average is not a competitive advantage, especially for an organization (not necessarily this one) that competes on value of service and cannot compete on the cost of service delivery.

Market driven

The research helped ACNY prioritize quality improvement programs and allocate resources that focus on the critical issues that affect service quality. The organization is now more in tune with member expectations, making it a true market driven provider.

"If you're committed to quality, this type of research allows you to make sure you're focusing on what's important to customers. We plan on doing this over time to measure how we're succeeding in closing the gap and how our members perceive our service quality performance."