Editor's note: Harvey Magier is president of New York-based Consumer Outlook, Inc. This article is adapted from a presentation made last July before the ARF Copy Research Workshop.
Many advertisers initially test their commercials in a "rough" format-animatics, photomatics and ripomatics-to identify the most viable candidates and provide insights for executional refinements. But do "rough" commercials accurately reflect the impact of their "finished" counterparts? More specifically,
- Do finished and rough commercials elicit similar patterns of response?
- Are finished commercials more apt to create positive attitudes for the advertised brand or service? Are they more motivating, relevant or newsworthy than rough executions?
- To what extent does a finished format enhance perceptions of the commercial presentation? Is this format more entertaining, empathetic or likable than rough commercials?
To answer these questions, we compared the performance of rough and finished commercials using normative data derived from Consumer Outlook's Audience Response Profile (ARP) copy testing system.
Methodology
ARP provides a comprehensive profile of all possible reactions to the advertising message and commercial presentation. The technique consists of personal, one-on-one interviews among target audience respondents in geographically dispersed markets. Each respondent is shown two exposures of a single test commercial and asked:
- five ARP open-end communication questions, and
- to rate 52 standardized ARP statements via a six-point "agree strongly" to "disagree strongly" scale
The 52 statements are categorized into 11 discrete evaluative factors. Four of these relate to the critical advertising message:
- motivation
- relevant news
- reinforcement
- motivation
The remaining seven factors cover the array of reactions to the execution itself:
- entertainment
- empathy
- realism
- uniqueness
- not complex
- good taste
- mood
This analysis is based on mean factor data developed from the ARP rating battery. Comparable samples of finished and rough commercial tests were drawn from Consumer Outlook's ARP normative base. They were matched in terms of equal representation across ten broad product categories (e.g., household products, beverages, services, etc.) and further balanced by client, within category. This yielded matched samples of 186 commercials per format-each representing approximately 10,000 respondents.
Findings
Factor correlations
Do the inter-relationships between factors differ by commercial format or are they relatively constant? If they differ substantially, how can rough commercials serve as a valid barometer of consumer reaction?
The following chart shows the correlation of the factor "relevant news" to the remaining ten ARP factors, for finished and rough commercials.
 |
For both finished and rough commercials, "reinforcement" is the factor most highly correlated to "relevant news"- "uniqueness" is the lowest. In general, the executional factors exhibit a lower correlation to relevant news than do the product factors. However, across all factors, there are only minor differences in correlation between rough and finished commercials on relevant news. This pattern holds true when we compare each of the factors versus each other.
In summary, the patterns of factor correlations for rough and finished commercials are similar. Thus, the relationship between factors is relatively constant, and not affected by commercial format. However, while consumer patterns of response are similar, this does not necessarily imply that they react as favorably to both formats.
Relative ratings
Are the levels of consumer response similar for both formats or do finished commercials engender more favorable reactions? The following chart compares the percent mean differences for the product related factors-those measuring consumer reaction to the strategic product communication. The percent mean differences are calculated by dividing finished mean ratings by rough mean ratings.
Bars to the right show that finished commercials received higher mean factor ratings than rough commercials; bars to the left show that finished commercials receive lower mean factors ratings than rough commercials. The broken lines at plus and minus seven percent show the difference generally needed for significance at the 90 percent level of confidence for a typical sample size of 100 respondents.
 |
Finished commercials received higher rating levels than rough on all product factors. However, these differences are not significant. Finished and rough commercials generate almost identical mean rating levels on the factors of credibility and relevant news. Importantly, relevant news has been found to be the one factor most predictive of on-air recall levels and persuasion scores. Thus, while finished commercials do enjoy a marginal advantage to rough, this advantage is relatively narrow and not significant on the critical product factors.
Now, looking at the executional factors-that is, the factors which relate to commercial presentation-we see a wider variation than found for the product factors.
 |
Finished commercials enjoy a significant advantage on the emotionally related factors of "mood" and "entertainment," plus a marginal advantage on "empathy." The presence of "real" people and the ability to show live action in finished commercials probably accounts for this enhanced consumer emotional involvement.
The fact that rough commercials are more apt to be considered unique is not surprising, given the consumer unfamiliarity with this format. Yet, the rough format does not appear to influence perceptions of commercial "realism."
Conclusions and implications
Based on these findings, we can conclude that:
- Rough and finished commercials evoke similar patterns of consumer response-that is, virtually identical factor correlations, within format.
- Finished commercials do not create meaningfully stronger positive attitudes toward the brand-and are at parity to rough on the critical factor of relevant news.
- Finished commercials are significantly more emotionally involving and entertaining than rough executions.
- Although rough commercials are perceived as somewhat more unique than their finished counterparts, they're viewed as equally realistic.
There are a number of implications suggested by these results:
- Rough commercials can be used with confidence to determine consumer reaction and response to the strategic product message; the incremental value of a finished format is marginal in terms of its effect on product-related perceptions.
- The fact that finished executions are more emotionally involving than rough does not usually create a problem-clients typically test pools of commercials with similar formats. However, in those instances where rough executions must be evaluated vs. finished, consideration should be given to weighting the rough commercial data to reflect these differences.
- Rough executions relying heavily on emotion or imagery should be in a format the closely approximates finished, e.g. ripomatic or photomatic. This is particularly true for executions using "star" talent or with a strong action orientation.