An integrated approach

Editor’s note: Jim Slevin is manager of field quality engineering at Advanced Micro Devices, a supplier of integrated circuits based in Sunnyvale, Calif. John Chisholm is president of CustomerSat.com, a Menlo Park, Calif., customer satisfaction measurement and market research firm.

Advanced Micro Devices (AMD) is a global supplier of integrated circuits for the personal and networked computer and communications markets. AMD produces processors, flash memories, programmable logic devices, and products for communications and networking applications. Founded in 1969 and based in Sunnyvale, Calif., AMD had revenues of $2 billion in 1996.

Prior to 1997, AMD had for many years conducted its annual customer satisfaction, loyalty, and value survey of over 200 of its largest customers through face-to-face and phone interviews and postal and fax questionnaires. In search of a more streamlined approach for its 1997 survey, and to make providing feedback more convenient for its customers, the company turned to CustomerSat.com, a Menlo Park. Calif., firm specializing in measuring customer satisfaction and conducting market research using E-mail and the Web.

The result was, to the best of our knowledge, the first annual, worldwide customer satisfaction survey by a Fortune 500 company whose using the Internet as its primary medium.

Why the Internet?

The Internet offers many advantages for customer satisfaction measurement (CSM). Survey results are typically available in a few weeks, with early results available in days or even hours. Customers can complete questionnaires whenever they choose from wherever they choose. Keyboard, mouse, and computer screen make answering surveys easier than handwriting responses, especially open-ended ones, and faster than being read a questionnaire over the phone. Additionally, human interviewer influences that can bias responses are eliminated. Finally, the costs of Internet survey deployment, response tabulation, and capturing verbatim open-ended responses are lower than conventional interviews.

For AMD, the key issue was how many of its customers would be able to respond to an Internet survey. Before all else, Internet-based CSM requires that the customers who have access to the Internet are representative of all of a business’ customers. More and more industries are meeting this requirement. In software, computers, networking, technical publishing, semiconductors, and graduate education, using the Internet for CSM and other survey research already is, or is rapidly becoming, feasible. For Internet-based businesses and services, such as securities trading, information retrieval, on-line gaming/entertainment, and other on-line services, customer access to the Internet is implicit. For internal customer surveys, where many or all of an organization’s employees share a corporate E-mail system, an Intranet survey is practical even if workers have no access to the external Internet. But Internet-based CSM is still several years away from being practical for most mainstream, non-computer-oriented consumer products and services.

Because AMD’s customers are almost exclusively involved in the manufacture of electronic equipment, we believed that most of them had Internet access. Consultation with AMD’s field sales organization confirmed that a list of E-mail addresses for most of AMD’s largest customers could be assembled from various corporate databases. E-mail addresses for other customers, if required, would have to be collected by phone or fax.

E-mail or Web?

The next major question was whether to conduct the worldwide survey using E-mail or the Web. Each approach has advantages and limitations. E-mail surveys use pure text (ASCII) to represent questionnaires, and can be received and responded to by anyone with an Internet address, whether or not they have access to the Web. Respondents edit the messages, typing in characters at designated places to answer either closed-ended or open-ended questions, and click on "reply." Respondents don’t have to be connected to the Internet while completing the survey; they may download the message into a laptop and complete the survey off-line.

E-mail surveys also have limitations. The limited intelligence of ASCII text cannot keep a respondent from, say, choosing both "yes" and "no" to a question where only one response is meaningful. Responses cannot be validated until the completed survey has been E-mailed back and received by the surveyor, which may be hours after the respondent has completed the survey. Skipping instructions (e.g., "If no, go to question 34") must appear explicitly, just as on paper. These factors can reduce the quality of data from an E-mail survey and can require post-collection data cleaning.

A final limitation is that some PC E-mail software products (such as Lotus cc:Mail) limit the length of the body of an E-mail message to 20,000 bytes of text, or anywhere from 30 to 100 questions, depending upon the amount of text in each question. If some AMD customers were using one of these E-mail products, they would not be able to receive and respond to an AMD E-mail survey if it exceeded the 20,000-byte limit.

Web advantages

In contrast to E-mail surveys, Web surveys use hypertext markup language (HTML), the language of the Web, and are posted on a Web site. Web surveys have several advantages over E-mail surveys. First, radio buttons, check boxes, and data entry fields, which are possible in HTML but not in ASCII text, keep respondents from selecting more than one choice where only one is meaningful, and from otherwise typing where no response is intended by the surveyor. Second, skipping can be performed automatically and implicitly based on a respondent’s answers, rather than having to be spelled out as instructions to the respondent. Third, responses may be validated as they are entered. Finally, additional survey elements, such as graphics, images, animations, and links to other Web pages may be integrated into or around the survey. These factors make completing the survey faster and more interesting, and result in higher quality data.

For survey research to be meaningful, surveyors must be able to control the pool from which respondents are selected, and ensure that respondents do not respond multiple times ("stuff the ballot box"). These requirements are met by E-mail surveys, whose distribution is controlled by surveyors and which can be encoded, if desired, to associate returned responses with their corresponding outbound E-mailings. To accomplish the same with Web surveys, E-mail invitations are used. E-mail messages containing the Web address (Uniform Resource Locator or URL, usually in the form http://www.company.com/etc.) of the survey page are sent to respondents, who either click on the URL, or copy and paste it into their Web browser, to view the survey page. When used with E-mail invitations, a Web survey is best posted in a hidden location on the Web, so that non-invited Web surfers are unlikely to find it.

Lower response rates

A disadvantage of Web surveys relative to E-mail surveys is lower response rates. This is due to several reasons. First, in some organizations, either because of technical constraints or corporate policy, employees have access to Internet E-mail, but not to the Web. Second, simply getting to a Web survey requires a respondent to follow several steps: clicking on, copying and pasting, or typing in a URL, and waiting for a page to be downloaded from Web server to PC. These steps, not necessary for E-mail surveys, take time and confuse some respondents. Third, respondents generally need to be connected to the Internet while completing a Web survey; they may not be off-line, as with E-mail surveys. For all of these reasons, Web surveys tend to experience lower response rates than E-mail surveys.

The response rate that a Web survey with E-mail invitations will enjoy will depend in part on what percent of the respondents have Web-enabled E-mail clients. A Web-enabled E-mail client is PC E-mail software that allows a user to click on a URL in an E-mail message to view that page on the Web. In contrast, a non-Web-enabled E-mail client requires a respondent manually to copy and paste the URL from the E-mail message to their Web browser to view the Web page. Response rates will be higher for respondents whose E-mail clients are Web-enabled. All recent releases of E-mail software from leaders Microsoft, Netscape, and IBM/Lotus, as well as some 30-40 percent of the E-mail clients currently installed worldwide, we estimate, are Web-enabled. These percentages will rise as users of earlier versions of E-mail software upgrade to newer releases. As this happens, response rates for Web surveys will get closer to those for E-mail surveys.

(To measure the difference in response rates between E-mail and Web surveys, CustomerSat.com recently conducted a test as part of an attendee satisfaction survey for an Internet technology conference. The survey measured attendee satisfaction with the conference program, facilities, meals, registration, city location, and performance relative to other conferences. Approximately 70 percent of the respondents, who were highly Internet-savvy, used Web-enabled E-mail clients. Half of the 600 attendees were surveyed via E-mail, and half via Web surveys with E-mail invitations. The response rate to the 30-question survey, without benefit of pre-notification or reminders, was 24 percent for the Web survey with E-mail invitations, 30 percent for the E-mail survey. We believe the Web survey response rate would have been lower had not such a high percentage of the respondents had Web-enabled E-mail clients.)

Unwieldy for some

The AMD research used a Web survey with E-mail invitations. The most important factor in the decision was the length of the survey - over 90 questions - which would have made an E-mail survey impossible for some respondents and unwieldy for others.

Being sensitive to response rate issues, we decided to offer respondent incentives for the AMD survey. With an Internet survey, a random drawing for one or more valued items is the simplest form of incentive to manage. That way, only a limited number of incentives need to be awarded, rather than one for every respondent. We wanted the incentive to be attractive to the design engineers, purchasing agents, and other professionals who would be completing the survey, as well as business-oriented. We decided to award two US Robotics PalmPilot hand-held personal organizers, each a $300 value.

Because customers were accustomed to conversing about AMD products and technology in English, we felt comfortable conducting the survey in English worldwide. To assure customers that individual responses would be kept confidential and that data tabulation would be performed objectively, the survey was hosted on CustomerSat.com’s Web site. (A copy of the survey questionnaire can be viewed at http://www.CustomerSat.com/amd.htm.) Invitations were sent out, primarily by E-mail, to the customers over a two-week period from AMD’s director of customer quality systems.

Questionnaire design

To determine the key performance attributes that the questionnaire would measure, pre-survey interviews were conducted with a sample of AMD customers. The survey covered six major AMD product lines, including microprocessors, non-volatile memory, networking, communications, and programmable logic products, and embedded processors. To make answering the questionnaire easier for respondents, performance attributes were grouped into five categories:

  • product quality and effectiveness
  • ease of doing business
  • support quality and effectiveness
  • sales quality and effectiveness
  • value and pricing

For marketing purposes, we were interested in what attributes customers believed were most important as well as what attributes actually drove their overall satisfaction and loyalty. So we decided to ask the importance of each attribute explicitly, as well as derive the importance using factor and regression analysis, in order to compare the two. Demographic variables used for cross-tabs included size of customer organization, primary market, type of customer (manufacturer, distributor, or reseller), respondent job function, length of time product(s) has been used, and geographical location.

Results in 30 days

It took less than 30 days to achieve over 200 responses, our targeted number, to the 95-question survey. Respondents included vice presidents and directors of AMD customer companies. During the 30-day period, reminders were sent to respondents as needed by E-mail and voice mail.

In the invitations to the survey, customers were given the choice of responding by Web, E-mail, fax, or postal mail. Ninety-three percent responded via the Web; 7 percent responded by other means. Most of those who responded by means other than the Web were outside of North America, where access to the Web is less widely available. Others worked for companies that restricted or blocked access to the Web. To these customers we either faxed a hard copy of the questionnaire, or E-mailed it as an HTML attachment.

Customers were enthusiastic about the Web survey. Several customers commented on the speed with which the Web survey could be completed. Face-to-face or phone interviews would have required 45-55 minutes of a customer’s time; the Web survey could be completed in 12-15 minutes. One of the survey questions asked whether the customer would be willing to be contacted for follow-up probing, if required. Combining the Internet with selective, follow-up probing by phone - pinpointed to the respondent and question - provides the advantages of phone interviews at a lower cost.

"The Web survey process allows us to gather vital data from customers in much less time and at great savings to both AMD and our customers," says Bruce Hicks, AMD director of customer quality systems. "Over time, more and more of our customer satisfaction measurement will shift to the Internet."