Search no further
Editor's note: Richard Kottler is new product director with Quantime Corp., a New York-based provider of survey research software and support services.
The advantages and disadvantages of using the World Wide Web to conduct survey research have been debated ever since the Web became acknowledged as a viable business tool. While speculation continues and skeptics abound, panels of Internet-connected respondents are growing throughout the U.S., and the marketing value of the data collected is increasing. Many researchers are discovering firsthand the realities, opportunities and challenges of Web-based research.
The additional challenge of ensuring that the technology used to collect data on the Web is the most appropriate for the medium and its users falls to marketing research software providers such as ourselves. The past year has proved a learning experience for both Quantime and our customers in this regard. We find the results encouraging.
The roots of Web-based survey technologies
Remember that the Internet is more than the Web. The Internet, dubbed the Information Superhighway just a few short years ago, can be thought of as the network which carries the information. Our firm has been using the Internet since 1982 for file transfer, for E-mail and for getting information via news groups and the like. However, in the same way that it took a reliable and graphical interface like Windows Version 3 to bring computers out of the domain of data processing specialists and on to the desktops of all, so the World Wide Web - again a graphical and very easy to use interface - has brought the Internet to the masses.
The development of the Internet and the Web laid the foundation for Web-based marketing research. The challenge for marketing research software developers was to adapt our existing software used for computerized interviewing (CATI and CAPI) to this medium. The final test was to implement this new technology in a real-world situation. This opportunity was brought to us through one of our customers, U.K.-based Continental Research, in partnership with Yahoo!
Yahoo! A case study
Yahoo! Inc. is an Internet media company that offers a network of globally-branded properties, specialized programming, and aggregated content distributed primarily on the Web, serving business professionals and consumers. As the first on-line navigational guide to the Web, www.yahoo.com is the single largest guide in terms of traffic, advertising and household reach, and is one of the most recognized brands associated with the Internet.
For Yahoo! the Web was an obvious means of surveying customers. As an electronic medium, the company was already in a position to supply exact traffic figures to companies advertising with it. But as a sales-driven business model, its ideal was to empower its advertisers with more accurate statistics as to who is visiting Yahoo!'s sites, and provide detailed profiles of the individuals who actually make up its audience.
Yahoo!'s European sites welcomed 70 advertisers during the first quarter of 1997, and Yahoo! Inc. recently announced that IBM Corporation, one of the top three Web advertisers, has selected the Yahoo! Network to launch the world's first global multilingual Internet advertising program. Other major advertisers with Yahoo! include British Airways, Opal, Nescafé, Peugeot and Karstadt. This therefore underlines the measurement of Yahoo!'s audience as important for growth of its industry and advertising on the Internet.
The challenge
Last year, Yahoo! requested marketing research companies to submit a proposal for the analysis of basic demographics of business and consumer users of their sites in Germany, France and the U.K. The objective of the study was to measure motivation of usage and point of primary Internet access. Yahoo! commissioned the U.K.-based agency, Continental Research, customers of Quantime since 1989. Although specialists in the field of advertising research, collecting data on the Web was a new concept to Continental. Continental's James Burckhardt was aware Quantime was developing a product dedicated to Web research and asked us to come on board.
So earlier this year, we joined forces with Continental and together devised a two-stage program designed to provide Yahoo!'s advertisers with audience qualification figures. Stage one was fielded between April 1st and April 15th. The purpose was to analyze the basic demographic profile of business and consumer users visiting Yahoo! France, Germany, and U.K. and Ireland and to measure motivation for usage and point of primary Internet access. It was essential to Yahoo! that the survey and any direct correspondence was carried out in the respondent's own language.
Respondents were also asked to supply their E-mail address for re-contact at stage two, which was to be a much more in-depth survey. Our primary methodological considerations at this stage were recruiting respondents to participate in the survey, persuading them to complete it and then ensuring that we were getting the best information we could. The fact that stage one of the survey attracted over 10,000 completed responses from the three countries in just over two weeks is an excellent sign that our objectives were being met.
Collecting the data
The first of the Yahoo! surveys consisted of 10 questions which probed respondents about their media preferences, education, age and expenditure patterns. Our major design objective in developing the Web software used in the Yahoo! survey was to maintain compatibility with our existing CATI and CAPI package, Quancept. Since the same scripting language is used, our Web surveys can now have exactly the same logic as CAPI or CATI surveys. Complex routing and randomization procedures can be built into the text, ensuring the data collected is consistent. Furthermore, the answers to previous questions can be used in the text of subsequent questions - customizing the survey for each respondent and encouraging their cooperation.
Most HTML surveys typically do not check for missing responses, and the most sophisticated of those that do often present the whole survey once again for completion. Since the majority of Web surveys remain standard HTML forms, they have the same limitations as paper questionnaires. Simple to complete, they tend not to have any complex routing and are quite short.
We believe the Web is about interactivity, adapting the questionnaire to the respondent as he completes it. This approach provides control in guiding the respondent to complete the survey completely and consistently.
An argument against this method is that viewing and answering a Web survey page by page could be slow, depending on the speed of the connection and Internet traffic. To lessen this in the Yahoo! survey, we presented questions that did not require routing on a single screen to minimize the time required to complete the survey.
Despite such tricks, we still found that around 10 percent of respondents who started the questionnaire failed to complete all the questions. There could be any number of reasons for this (boredom, connection problems, impatience), but since the cost of these lost interviews was close to nothing, it made little difference. In stage two, where we requested that the respondent sign-in with their E-mail name, we were able to offer the possibility of restarting the uncompleted survey at a later time, and picking up neatly where they left off. The result of this is that despite the stage two questionnaire being substantially longer the drop off rate fell to around 5 to 6 percent. This was somewhat assisted by the exhortation to complete contained in a personalized E-mail and the opportunity to win one of five hand-held electronic organizers.
In line with other studies of Web usage, 80 percent of the respondents were male but, surprisingly, around 60 percent were employed and over 35 percent were between 25 to 35 years old. The study also exploded a general myth that the primary Web users remain businesses; although around half use it for both business and personal, of those who stated exclusively one or the other, twice as many Web users in the survey used it solely for leisure and personal reasons.
Stage two - The in-depth study
While stage one of the exercise invited anyone who clicked on the advertising banner to complete the initial survey, stage two was conducted among those who left a valid E-mail name in stage one and agreed to participate further. These respondents were sent an E-mail informing them of the survey location on the Web. The survey was much longer than the first stage, consisting of a series of in-depth lifestyle questions. Because these respondents are known to us, we could ask them to sign in and in doing so, we could very accurately measure the response rate. We could send out reminders, if necessary and ensure that individual respondents only completed the survey once. In fact we achieved our target sample size within one week of sending the E-mail notices, and no reminders were necessary. On Thursday, May 22, 3,163 E-mail messages were sent out during the day. Ten days later, we had received a total of 2,263 hits, which represents a potential 72 percent response rate. Also note that Monday, May 26 was a public holiday in the U.K.
Continental Research will be analyzing the data in depth and reporting on the results later in June 1997.
For a third step, Yahoo! will put together large cross-media surveys to provide potential advertisers information about the quality of audience who access certain Web sites. Using audience qualification tools like MRI indices, similar surveys are run by Jupiter Communications in the U.S., but so far there have been no Internet-wide audience surveys conducted.
Sensible and viable
Results of the Yahoo! survey indicate the Web is a sensible and viable vehicle for surveys of Internet users. As shown by the results so far, we can achieve excellent response rates for a fraction of the cost of traditional methods. But what about researching issues that are not technology related? Is using panels to conduct consumer research the only viable method? The answer, in my view, is a resounding yes, but I believe continued research into the different survey techniques is required before we can proclaim the Web a substitute for other data collection methods. However, we intend to continue to experiment, to learn and to adapt the very real opportunities offered by the Web, where appropriate, for our customers throughout the world.