Editor’s Note: Wes Breazeale is an associate at MSInteractive, a Portland, Ore., manufacturer of dial-based measurement hardware. This article is an update of a paper originally presented by Friea Crocker at the International Conference on Computer Assisted Survey Information Collection in December 1996 in San Antonio, Texas.
As researchers attempt to delve into the consumer’s mind, they have encountered significant barriers and frustrations. Whether these consumers are the targets of advertising or students in a lecture hall, there is an ongoing desire to know what message will reach them most easily, what methodology will prove most effective in communicating the presenter’s message. Researchers struggle not only to understand a consumer’s final decision, but also the thought process that leads to it. Nowhere is this more evident than during the testing of continuous stimuli such as TV or radio advertisements. During each moment the subject may be experiencing a different reaction or emotion.
To provide a representation of a group’s moment-by-moment reactions, many researchers seek alternative methods of collecting information. One method that has gained popularity and media attention is the use of electronic group measurement systems (sometimes called audience response systems). While experiencing continuous stimuli like a TV advertisement, respondents use a small dial to continuously rate the material using a 0-to-100 scale. A remote interface connected to a PC collects data from the dials while software running on a PC aggregates the data and displays a real-time moving line graph. Variations in the line graph illustrate positive and negative moments and key turning points. This display can be overlaid on the test video so the graph can be viewed later by the client or shown to respondents to stimulate group discussion about high and low points in the video.
Developed decades ago
Although electronic group measurement systems began attracting widespread attention in the past decade or so, the original concept for interactive group testing was developed decades ago and used in advertising research as early as the 1930s. Frank Stanton of CBS and Paul Lazarsfeld of Columbia University developed one of the first systems for measuring continuous audience response. The Program Analyzer was originally designed for radio testing and its usage evolved as CBS expanded into television broadcasting, where it was used to gauge the success of potential CBS pilot episodes. Respondents held large plastic knobs in each hand, and were instructed to press either a red button on the left knob or a green button on the right knob depending on their reaction to the television show. The results of 10-12 testing sessions would be combined and summarized as red and green curves on a continuous strip of paper.
The entertainment industry has continued to apply audience response systems in the design of new programs and films. In the 1960s, Columbia Pictures created Audience Surveys Incorporated (ASI) to conduct research and quantify the value of potential TV pilots. Pierre Marquis expanded on existing applications of audience response systems and, in 1965, introduced the Hollywood “Preview House.” Designed with built-in dials at 200 of the 400 theater seats, the theater presented an opportunity for researchers to gather immediate audience reaction from a broad sample of respondents. Instantaneous Response Dials wired into the individual seats translated audience responses into interest curves viewed in an upstairs control room.
The Preview House was first used primarily for testing movies and pilot programs. According to Gerald Lukeman, chairman of ASI Market Research, almost every ABC and NBC television show aired from the 1960s to the beginning of the 1980s was first subject to extensive testing in the Preview House. Because the dial system could provide second-by-second information, using the dials for the testing of commercials was also commonly a part of the TV show evaluations.
Portable systems
Technological advances in microcomputer development made possible the first portable system in the early 1980s, introduced by PEAC Research in Toronto. Portability allowed for greater flexibility in testing locations and, for the first time, researchers were not restricted to wired theaters and permanent rooms. The PEAC system stored participant reactions in a data buffer located in the hand-held units. Later, collected data would be downloaded into a central processing unit for analysis. These early systems, while portable, were bulky, wired systems that required several hours before and after the group session for preparation and tear-down.
Today, a wireless electronic group measurement system can fit into a piece of carry-on luggage and requires less than an hour to set up. These days there are several respondent input devices available for gauging continuous audience response. Subjects can push keypad buttons, move a slide, or turn a dial to record their moment-to-moment reactions to any continuous stimuli. Some input units combine the keypad function with the dial or slider. Each variation has its advantages and disadvantages, but dial-based systems tend to be the most easily used for testing continuous material like audio and video. Most of the dial- or slider-based systems use a 0-to-10 or 0-to-100 scale and can record even minor variations in reaction. Depending on what sort of software is being used with the system, responses can be displayed in a variety of ways. Particularly useful is the ability to display the data in real time as a line graph for the group as a whole, with additional lines representing particular subgroups or segments defined by answers to earlier categorical questions. These subgroups are typically based on demographic characteristics but can be anything from usage patterns to how respondents rate products of interest.
The capabilities of these systems are also much increased and the key to getting the most out of the system is having a software package that may take advantage of all available data. Many electronic group measurement systems use software that creates data files compatible with SPSS, Lotus 1-2-3 or Excel. More recently, companies are offering the ability to download directly into an HTML format for easy posting to the Web, or directly into a PowerPoint format for quick presentation displays.
Advances in technology have also brought the cost of these systems down, even as capabilities have increased. In 1990, a 50-dial or -keypad wired system might cost up to $60,000. Today many wireless systems this size sell for $15,000 to $22,000, with a basic wired systems typically much less. Compared to their old wired counterparts, the new wireless systems tend to be both more reliable and less complex to set up.
Diversified applications
The newfound flexibility of electronic group measurement systems allowed for a diversification of research applications. No longer were the systems confined to the large entertainment studios who could afford the investment of a fixed installation. While the systems continue to be widely used in the entertainment and advertising industries, new uses were found when the systems became more mobile.
One such use was with jury research. Electronic group measurement systems have proven to be quite helpful in evaluating the effectiveness of a potential presentation to a jury as well as in the selection of the jury itself. Attorneys present their cases to mock juries for insights into personal mannerisms or particular aspects of their presentation that might prove to be more or less persuasive. In addition, in tracking responses by demographic segments, jury researchers are also able to quickly create a profile of an ideal juror for each particular case. In a similar way, audience response systems can also be useful to professional speakers and trainers. Their aims are generally similar to that of a lawyer presenting a case - to either provide some information to an audience or to persuade them of the validity of a certain point. The effectiveness of various presentations and programs may be quickly measured, and listeners tend to be more focused on the presentation itself, since the systems allow for easy audience participation.
Audience response systems have also been used for a wide variety of non-commercial purposes. PBS has used a dial-based system to maximize results from fundraising campaigns. To get a feel for the opinion of the public at large, a keypad system was used in Alaskan town meetings, allowing hundreds of residents to vote on the allocation of state oil revenue. During the development of a university rape-awareness program, administrators used a dial-based system to display the reactions of a mock jury. A New Zealand research firm has used a dial system to gauge the success of a nationwide anti-smoking campaign. The variety of uses for an electronic group measurement system is staggering.
Some of the most well-publicized uses of audience response systems are in the political arena. Continuous research made its first appearance in politics during the 1960 Kennedy-Nixon debate and reappeared in the mid-seventies. Pollster Fred Steeper of Market Strategies used a dial-based electronic group measurement system to analyze voter reaction during the 1976 debates between Gerald Ford and Jimmy Carter. Electronic systems have provided strategic information in virtually all subsequent major campaigns, including the last three presidents and both Al Gore and George W. Bush. During the planning stages of Clinton’s 1992 campaign, several proposed platforms gained little response from traditional focus group testing. In frustration, pollster Stan Greenburg combined electronic group measurement systems with his existing research methods. The moment-to-moment information highlighted several comments as highly appealing; these were expanded and incorporated into Clinton’s political message. Although Clinton has been described as the “most research-driven president in history,” almost every major politician in the past two decades has used dial or keypad testing for research, and most professional political advertisements have first been tested through focus groups and interactive research. While the popularity of other types of research has declined, focus groups and interactive testing have steadily increased as the political research methods of choice.
Electronic group measurement systems have various academic applications as well. Since 1971, Cornell has used a system for student response during lectures. Rutgers University designed its own keypad response system in a large amphitheater to allow student feedback and participation during introductory physics lectures. The system has been used in nine classes since the beginning of 1996, and more colleges and universities are adding systems to various classes. In a recent questionnaire, 95 percent of students reported that an electronic response system had been helpful for increasing learning during a lecture-based class. In addition, other studies have shown that utilizing an electronic student response system not only increases the students’ level of attention during a lecture, but it also helps reduce the variation in test scores from student to student.
Of course, the entertainment industry still lives and dies by electronic audience response testing. From the evaluation of advertisements to attitudes towards various personalities, electronic measurement devices provide a wealth of information. They can offer suggestions about which spokespeople and images are most likely to evoke positive reactions from consumers. Information gathered from audiences offers major studios feedback about character development and plot design in both television programs and feature films. Each year the major TV networks conduct extensive testing of pilot shows. Several research companies use the systems for insight in the development of television news and radio formats. Audience response systems have even used for entertainment purposes in and of themselves. A keypad audience response system was used a decade ago for instant audience voting on the widely-televised America’s Funniest Home Videos, and a system is currently used for the popular Who Wants to be a Millionaire?
Achieving potential
Although electronic group measurement systems are no longer in their infancy, the technology is just now achieving its full potential. These systems can provide an accurate representation of consumer response throughout a continuous stimuli. Using a continuous response system with other research methods provides an understanding of audience responses to the stimuli both as a whole and during individual turning points. And while an electronic group measurement system may not be able to read the minds of audience members, they have allowed researchers to gather more data than was previously available. In the ongoing effort to learn more about consumers, audience response systems are a valuable tool.
References
Allen, Mike (1996), “The Applause-O-Meter Still Thrives in Politics,” The New York Times, August 26, C5.
Fenwick, I., Rice, M., (1991), “Reliability of Continuous Measurement Copy-Testing Methods, Journal of Advertising Research, 31, Feb/Mar, 50-59.
Hughes, G. David (1992a), “Realtime Response Measures Redefine Advertising Wearout.” Journal of Advertising Research 32 (3), 61-77.
Hughes, G. David (1991), “Diagnosing Communications Problems with Continuous Measures of Subjects’ Responses: Applications, Limitations, and Future Research,” in Current Issues and Research in Advertising, James H. Leigh and Claude R. Martin, eds. Ann Arbor, MI: University of Michigan.
Hughes, G. David and Richard D. Lennox (1990), “The Reliability and Validity of Realtime Response Measures,” listed under the title “Realtime Response Research: Construct Validation and Reliability Assessment,” in Enhancing Knowledge Development in Marketing, William Bearden et al., eds. 1990 AMA Educators’ Proceedings. Chicago: American Marketing Association.
McTague, Jim (1995), “Brain Pickers,” Barron’s, May 29, 25-26.
Newsweek, (1996), “Jazz for Drive Time” May 20, 49.
Newsweek, (1992), “‘Manhattan Project,’ 1992,” Nov/Dec Special Election Issue, 40-56.
Sanoff, Alvin P. (1995), “The Consulting Game,” U.S. News & World Report, Sept 18, 119-122.