Researcher as teacher
Editor’s note: Chuck Young is CEO of Ameritest, an Albuquerque, N.M., research firm.
When I first started in this business, a wise old researcher told me that a researcher has two jobs: first, to learn something useful that your clients didn’t know before; second, to teach them what you found out. Over time, I learned that the second job is the harder of the two. It is also the more important, because it is the key to making sure that the research you do actually makes your clients smarter and gets used.
This is particularly true of advertising pre-testing. No form of research is more fraught with barriers to learning. The issues raised by pre-testing generate high anxiety for everyone around the conference room table with a vested interest in the advertising.
Confusion and anxiety do not, as a rule, form an ideal emotional climate for the learning process. Clarity and calmness are required.
How do we move beyond these limitations and become the teachers our clients need us to be? We must provide our audience with a shared mental model.
To borrow from Peter Senge’s well-known book on learning organizations, The Fifth Discipline, “The effectiveness of a leader is related to the continual improvement of the leader’s mental models.” In other words, the decisions clients make based on pre-testing research are as much a function of the mental models they have about how advertising works as they are of the information that you provide them.
Don’t think your clients can all agree to use the same mental model? Think again.
A brief history of advertising pre-testing
First, let’s remember why our clients all have different mental models. Dating back to the early days of television, the first widely used pre-testing measure was Burke’s Day After Recall Score, which said effective advertising should leave some kind of memory trace in the consumer. Unfortunately, after many years of empirically trying to correlate recall scores with sales results, a number of advertisers, such as Procter & Gamble, concluded that recall was missing something important. So, researchers searched for something else to predict sales. In the ’70s, pre-testing research shifted its focus to measuring motivation, such as the Research Systems Corporation’s ARS measure of persuasion.
 |
In the ’80s, another pre-testing company, ASI, now IPSOS-ASI, found that recall could be better understood when its two component variables were separated: the attention-getting power of the commercial execution and the linkage between the brand and the commercial.
Meanwhile, other researchers argued that the likability of the commercial was key - a result empirically confirmed over a decade ago by a famous Advertising Research Foundation validity study1.
Most researchers also agreed that communication of a strategic selling proposition was the key to effective advertising, a point of view that continues to sell a great many focus groups to this day.
Creatives, who appear to have different mental models of advertising than researchers, have always intuitively felt that the entertainment value of a television commercial matters, that it’s important to be fresh and different to stand apart from the crowd.
What about emotion? Emotion sells. So, on a parallel track, advertising agencies, like the Leo Burnett agency, developed complex methods of coding and analyzing the verbatims from open-end questions and constructed batteries of diagnostic ratings statements to profile viewer response to commercials on multiple dimensions.
And finally, a number of researchers believe that pre-testing shouldn’t just be copy-testing. After all, we are attempting to describe the consumer’s viewing experience. These researchers experiment with non-verbal techniques: brain waves, galvanic skin response, voice pitch analysis, and picture sorts.
So, like blind men arguing about elephants, the debate goes on to this day. No wonder clients are confused and creatives are skeptical!
Is the real question, “Which of the above measurements is the correct one?” Consider this: many smart people have been working on this problem for many years and each theory is probably right to some degree. From a teaching standpoint, the real problem may be one of synthesis and interpretation, not measurement.
If the goal is to help clients to make smarter decisions about their advertising, and therefore become leaders in their business categories, researchers should ask themselves: “How do I fit these different ways of measuring the advertising experience together into a more complete and intuitive description of the advertising?”
One answer to this question is the Ameritest Advertising Model, shown in Figure 1.
A heuristic model
In this heuristic model, information is arranged in a hierarchy that bridges the divide of report card systems and diagnostic systems.
At the top is what pre-testing is supposed to predict: in-market sales results. One level down are the evaluative measures that provide the report card portion of the analysis. Two levels down are the diagnostic measures that are correlated with, and therefore explain, the evaluative measures above.
The arrows in the model highlight the primary relationships between the different variables measured and hence provide a road map for interpreting the data.
Both report card and diagnostic systems use evaluative measures in an attempt to explain why a commercial is or isn’t working. Examples of evaluative or report card systems would be ASI and ARS; examples of quantitative diagnostic systems would be Diagnostic Research, Inc. (DRI) and our firm.
You will recognize the report card variables as the fairly conventional ones discussed in the history section of this article. Different systems measure these variables in somewhat different ways.
Essentially, the model says that for any commercial to be effective it must accomplish three things:
1) It must get noticed and attract an audience.
2) The audience must know who is sending the advertising message.
3) Once the commercial has the audience’s attention it must sell them something — i.e., motivate sales in the short run or at least create a positive predisposition for sales in the long run.
Other variables are important only insofar as they help to explain the variables of attention, brand linkage and motivation. For example, entertainment value is not important in and of itself but because it is an important predictor of attention.
The same is true of liking. On the face of it, it may be possible for a commercial to not be well-liked and still be effective. Wisk’s “Ring Around the Collar” campaign is a widely mentioned example.
But, intuitively, getting into the conscious mind of the consumer and selling the brand is always the bottom line for advertising!
Verbal versus visual
Also in Figure 1, we introduce visual diagnostics. To fully describe the total advertising experience - the aesthetic plus the semantic - it is necessary to complement the traditional verbal measures of advertising research with non-verbal measures.
The old adage that a picture is worth a thousand words is simply untrue. There is aesthetic information in a picture that cannot be put into words, just as you cannot fully articulate what you feel when experiencing a joyous piece of music.
So, how do we measure that which cannot be expressed in words? The non-verbal approach used in our system is based on a simple picture-sorting technique2.
In brief, during this part of the interview, respondents are either handed a shuffled deck of photographs or are shown the images on screen in a CAPI interview. These playing card-sized images, typically 10 to 30 pictures for a 30-second ad, are taken from the commercial itself. This deck of photos provides a natural vocabulary for respondents to use when describing their visual experience of the advertising, without using words!
 |
Armed with this new vocabulary, respondents first sort the pictures on the basis of recall: those pictures they remember seeing in the ad and those they do not. This data is collected to create a Flow of Attention graph for the ad (Figure 2).
Analysis of the pattern of recall, shown in the example in Figure 2, is an important predictor of commercial attention and brand linkage. Basically, this technique measures how viewers process the images in the commercial on a cognitive level. It tells us which images the viewers find meaningful, if they are able to follow the storyline, and if they are giving the brand the focus required for good brand linkage.
From a teaching standpoint, the Flow of Attention helps clients make the paradigm shift from thinking of the human eye as a recording device, like a camera, to thinking of it more as a computerized search engine that actively sorts through the information. Selective perception is the filter that alters an advertising message from what the agency intended to what the viewer actually understood.
Just as the Flow of Attention helps us to better understand attention and brand linkage scores, a Flow of Emotion graph helps us explain motivation scores. Using the same deck of images, the respondents sort the deck of pictures into six piles based on their feelings as they watched the commercial, from strong positive to strong negative response. This second picture-sorting exercise allows us to model the affective emotional response to the commercial.
Execution versus strategy
When clients are presented with unfavorable pre-testing results, they often wonder, “Is this telling me I have a bad execution or a bad strategy?” While historically the rule has been to use pre-testing to evaluate executions, not strategies, understanding the client’s strategy is important for interpreting pre-test results.
 |
When validating our system, a lack of correlation was found between the attention-getting power of an ad and its motivational impact, as shown in Figure 3. Attention and motivation are independent variables! This lesson should be at the top of your teaching list.
Knowing that an ad has the ability to break through clutter and attract an audience tells the client nothing about whether or not the audience will act on the ad’s message. Conversely, knowing that the ad’s strategic message has the power to motivate a viewer does not mean the ad will break through.
The recent frenzy of dot-com advertising provides us with many examples of self-indulgent ads that caught our attention but did not motivate us to act. An effective commercial has to do both.
This model reinforces this important idea. The left side is primarily about the advertising execution. The right side is about the strategic message being communicated. This is the creative yin and yang of advertising.
The model shows us that attention is a function of two primary aspects of the execution:
1) Entertainment - Does the execution entertain or reward viewers with an enjoyable or likeable or unique experience in return for the 30 or 60 seconds that they are asked to spend with the client’s message?
2) Flow of Attention - Is the execution a well-edited piece of film that captures and maintains the viewer’s attention over time, focusing their thoughts and feelings on the important ideas and images in the commercial at a pace that they can easily keep up?
Motivation is also a function of two dichotomous constructs:
1) Communication - How relevant, believable and brand-differentiating is the strategic promise you are communicating to your customer?
2) Flow of Emotion - How much emotion have you tapped into with the power of film to make your brand’s promise seem “larger than life” and even more compelling?
In short, the key to motivation is communicating a relevant idea in a dramatic way.
Teaching model
We consider our model a teaching model. It was designed to make our clients smarter users of pre-testing research so they can make better decisions about their advertising.
One of the funny things about models is that you never stop building them. After all, they are, by definition, simplifications of the real world that is always evolving. That was Peter Senge’s point.
There are other non-verbal techniques being developed that offer potentially interesting insights into advertising other than the approach we currently use. We are doing some experimental work with brainwave analysis that looks encouraging, for example.
But it is more than an academic exercise to build heuristic models. If teaching clients to use research to make better decisions about advertising gives them a competitive advantage in the marketplace, then improving your model over time is a way of sustaining that competitive advantage. This is the ongoing challenge for pre-testing research.
Notes
1 Russell I. Haley and Allan L. Baldinger, “The ARF Copy Research Validity Project,” Journal of Advertising Research 31 (March/April 1991).
2 For a more complete description and validation of the technique, see: Charles E. Young, “Creative Differences Between Copywriters and Art Directors,” Journal of Advertising Research 40 (May/June 2000).