Editor's note: Jim Strelchun is vice president of marketing with Ortek Data Systems, Inc., a Beaverton, Ore.-based maker of response measurement systems.
If you saw much of the TV news coverage of the presidential campaigns, there's a good chance you also got a peek at the results of some audience's blow-by-blow, graphed rating of a Bush-Clinton debate. The instantaneous results were provided by an electronic system that monitored and displayed the audience's voluntary responses during each second of the debate.
Audience or group response measurement systems (RMS), while known by a variety of names, all share a common purpose: find what the people in a gathering think about any issues or material presented and then display that information instantly. RMS products serve the insatiable demand for instant, actionable information found in virtually every application involving people in group settings, especially in market research.
For market researchers and consultants, RMS can be a tool for enhancing the accuracy, quality and speed of primary research applications. Their use is increasing as familiarity, knowledge and confidence in them grows. Such factors as improved cost, technology and usability have contributed to the growth. RMS, which three years ago cost from $15,000 to $60,000, is now available for $3,500 to $20,000. Technological improvements have made RMS more dependable and reliable, and most no longer require a programmer or "rocket scientist" to operate. Also, RMS is now capable of addressing more applications and providing real-time analysis.
How are they used?
In conducting a typical research session with RMS, the moderator might ask the respondents to answer a question automatically displayed on a monitor. Respondents are directed, for example, to select their favorite soft drink from seven choices. If they preferred choice one (e.g., Coke) they would press the "1" key on their keypad, for choice two (e.g., Pepsi), they would press the "2" key, etc. The results are electronically collected, analyzed and displayed on a monitor. Answers to virtually any categorical or intensity-scale questions can be gathered speedily. It's not uncommon to execute an "electronic survey" of 100 to 200 questions, with any size of group, within an hour or so.
Somewhat more sophisticated than the "discrete" response mode is the "continuous" response mode, typically used for testing ads and live events. Respondents indicate their changing emotional reactions to a presentation or other audio/visual stimulus on a second-to-second basis by using a variable motion mechanism such as a dial. The dial motion is interpreted by the computer as a positive or negative scaled response. Since all devices are simultaneously sampled once a second, the group's mean response is displayed as a moving line graph, which may be recorded and analyzed later with the group.
RMS with the continuous response mode
Aside from simply collecting responses to categorical questions, RMS can capture respondents' gut reactions to every moment of a commercial or presentation. A group's mean response is updated and displayed within a second of their turning the dial on their responders, as their feelings change. This continuous response mode is about the only way a commercial or live presentation can be instantaneously rated by a group without the problems of distraction (from stimulus) or bias.
In applications such as ad testing or presentation rating, a graph of the group's response is simultaneously merged with the video (stimulus) and recorded. Positive and negative scenes or lines, indicated by graph deflections, can be pinpointed immediately. The moderator then reviews the video overlay tape with the group and explores the reasons for their particular response at selected points.
Additional applications
When equipped with a responder unit in an "electronic focus group," respondents possess, in effect, their own private voting booth. Answers are non-verbal and anonymous, so RMS eliminates most of the typical group response problems that confront moderators, such as respondent reticence, group-bias, and group-think. When asked even the most sensitive or politically-charged questions, respondents can register their true feelings with complete confidence, without fear of recrimination or humiliation. No one respondent can bias or bully the group, and all answers carry the same weight.
The group discussion guide is generally integrated into the RMS host computer in the form of a question/answer list. If the list is completely executed by the moderator, an RMS can guarantee that no desired information will be overlooked or forgotten. An RMS will also assure a 100 percent response rate for all questions asked.
Because the moderator has more time to focus on selective probing, RMS sessions can be more productive. Less time is wasted on bias-control and trying to extract an answer from each respondent for every question. Questions and choices are presented non-verbally, so the challenge of moderator objectivity is minimized. Instant displays of results help the moderator determine when and where to probe for more information. Follow-up questions can be created and executed, allowing for a flexible, interactive process.
An added RMS benefit is that all respondent information is stored in a database. A summary can be printed out at the end of a session and results can be reviewed quickly on a computer monitor. Thus, the need to sort, organize and analyze many pages of notes is virtually eliminated.
RMS can facilitate product related research projects by prioritizing product features, for example. Respondents might be asked to rate a series of features or attributes relative to each other; RMS can then provide relative scores and rankings instantly.
Some moderators are reluctant to use RMS in a focus group because they consider it intrusive; some fear the client's involvement as results are being displayed. Users, however, find that RMS can actually enhance the group. It can improve the overall quality and enable the moderator to elicit more information from the group. With direct input through their personal responder units, respondents get more involved in the process.
RMS, with its ability to support hundreds of respondents, can provide an economy-of-scale for certain quantitative projects; researchers can often conduct multiple sessions, consisting of larger "macro" groups of 30 to 60 respondents. Such projects require only the added overhead of placing more responders in a larger meeting facility.
Clients like the feeling of participation with RMS and the fact that they can see the responses of the entire group as the session proceeds. If there's a printer, they can also depart from a session with something tangible: a complete summary of the group's responses.
RMS technology and operation
There are three major components in RMS: the "responder" device, the "data collection" device and the control/analysis software. The attributes of these components and how well they're integrated into a cohesive system determine the RMS effectiveness and value for a given research application. The degree of portability of a complete RMS depends upon the size and quantity of responders and the style of host computer (e.g., laptop).
The responder is the mechanism used by each respondent to register an answer to a question. It's probably the most critical component, since it links the respondent's answers, thoughts and feelings to the RMS and the researcher. Responder devices range from a simple, hand-held electronic dial or button to a full computer keyboard with a dedicated personal computer. Generally, the more complex and sophisticated the responder is, the more difficult to use and confusing it becomes for any but the most technically competent respondents. This is particularly important in dealing with a broad range of respondents, from children to seniors.
The simple, telephone style keypad mechanism is non-intimidating and proves to be most effective for accommodating discrete responses. Simple, easy-to-read feedback methods such as a bright numerical display help respondents see their selections. Also, for darkened rooms, a back-lighted keypad may be helpful to respondents. In terms of physical size, smaller hand-held devices provide more accessibility and privacy for the respondent.
For an RMS that supports the continuous response mode, special response mechanisms such as a dial are integrated into the responder. While keypads may also be used for this purpose, the dial is preferred by researchers because of less respondent distraction and faster response speed. It doesn't require respondents to look away from the test material and possibly miss a pivotal scene in a commercial. Respondents simply rotate the dial right or left to indicate a positive or negative feeling.
Since response data must be simultaneously collected from groups consisting of ten to even hundreds of respondents, it's necessary to have one or more electronic data collection units to perform this function. Each hand-held responder is typically cabled to a data collection unit, which "samples" each one in turn using a multiplexing algorithm. Ideally, the responders are sampled for data at least once a second. The collected data is then sent from the collection unit to a host, personal computer (PC), generally through the serial communications port (RS-232), common to most PCs.
Software for the control of RMS, if included, normally resides on the host PC under some version of DOS. As with any software, the number of functions performed, applications supported, and its ease of use will vary widely and will ultimately determine how much you can accomplish. RMS software is necessarily an amalgam of data acquisition, word processor, graphics, analysis and display processing functions.
Summary
RMS has proven effective for both quantitative and qualitative research due to its anonymity, privacy, electronic accuracy, speed and instant feedback. Respondent candor and objectivity can't be guaranteed, of course, but RMS can improve the odds. While it's the dominant solution in some applications where there is no better alternative (e.g., ad testing), its typical role in most other research applications is to enhance existing methods. For a growing number of researchers and their clients, RMS is becoming a mandatory rather than optional tool in many primary research applications.