Editor's note: Don Bruzzone is president of Bruzzone Research Co., Alameda, Calif. Paul Shellenberg is the firm's director of sales.

Once a year almost half the U.S. population sits down to watch the same program, the Super Bowl. But they are also watching scores of brand new commercials - which makes them the world's largest available test panel!

They think they are just watching a football game. So this enormous panel is perfect for research. There has been no forced exposure, no pre-sensitization and no strange surroundings to contaminate results. The commercials they are watching are produced by the best and the brightest in the business using immense amounts of money. Could you devise a better opportunity to learn what works and what doesn't?

With that in mind, our firm, Bruzzone Research Co. (BRC), started testing Super Bowl commercials five years ago. We've been doing it every year since. The following is a brief recap of what we have done and what we have learned.

People can forget advertising very rapidly. So we wait a week or two before checking to see if it is still having a measurable effect on them. When we do contact them we want to use something that gets through to virtually everybody, everywhere, on the first try. With today's busy lifestyles, voice mail and answering machines, the telephone was not our method of choice. Besides, we want to show them something. We do a lot of this type of ad tracking by approaching people at random in malls. But in this case we wanted to reach all types of people everywhere, and we wanted the number who decline to participate to be as small as possible.

We accomplished all these objectives by mailing questionnaires to a nationwide sample drawn from all households for which an address is available from either an auto registration or a telephone listing, six days after the Super Bowl. Each household receives questionnaires for 20 commercials (a sample is shown on facing page). Each page shows a photo board of the commercial with all references to the advertiser masked out, and BRC's full battery of diagnostic questions. They are filled out and returned by 37 percent of the recipients - a higher completion rate than you will normally encounter in either telephone or mall intercept interviewing.

For each of the 160 Super Bowl commercials aired during the 1992, 1993,1994 and 1995 games, reports were obtained from 302 to 365 individuals. At the time this is being written, we are in the process of conducting our 1996 research on 40 additional Super Bowl commercials, bringing the total to 200. Assuming the same response rate as prior years our database will soon consist of 66,000 individual reports on Super Bowl commercials. The results from the research have been voluminous but basically they show the value of the advertising on four levels:

First: How many noticed the commercials?

The best-remembered commercials from 1992 to 1995, and the percent of our respondents that recognized them were:

Pepsi: Shaq vs. Small Boy (1994)......................... 77 percent
Diet Pepsi: Ray Charles "Hearing" (1992)................77 percent
Columbia Pictures: "Few Good Men" (1993).............73 percent
Pepsi: Cindy Crawford "Boys on Fence" (1992)........67 percent
Chevy: Metal balls falling on pick-up (1994)..............66 percent

Not all Super Bowl commercials were this fortunate. Recognition of the bottom five ranged from 3 percent to 13 percent, showing a surprisingly wide range in recognition scores for commercials that had all appeared on the Super Bowl a few weeks earlier. Airing a commercial on the program with the biggest audience is no guarantee people are going to notice it.

Recognition provides the best measure of intrusiveness because it is the most accurate, complete and reliable measure of the number that noticed the commercial. It separates the people who noticed a commercial from those who ignored it, or were never exposed to it, so we can see if it had any effect on them. It shows if the communication process had a chance to start.

We also looked at a key measure of the information communicated by the commercials. How many remembered who they were for? Here the percent among recognizers ranged from 91 percent to 20 percent. The best at getting the name were usually the same that were best at getting noticed. But not always. Some commercials for jeans and credit cards did an excellent job getting noticed but not in getting the name across.

Second: How many were affected by the commercials they noticed?

The previous measures only show if the commercial had a chance to affect people. To find out if it actually did, two types of measures were used - likability and diagnostics - the two that were shown to be the best indicators of advertising effectiveness by the Advertising Research Foundation Copy Research Validity Study 1. BRC has been using both since 1985. Likability was the best of all measures, and since it applies to all commercials it lends itself well to the ranking of commercials. However, if people like a commercial but can't remember who it was for, it can't have an effect on sales. So, we combine the two in our second basic measure of advertising's impact: the percent of recognizers that knew who it was for and liked it. The commercials that performed best in this second pair of measures were:

Pepsi: Cindy Crawford "Boys on Fence" (1992)..........72 percent
Pepsi: Two drivers in a diner (1995).........................69 percent
MasterLock: Burglars (1992)...................................69 percent
Budweiser: Frog (1995)..........................................68 percent
McDonald's: Michael Jordan/Larry Bird (1993)............68 percent

Here there was an even wider range in the performance scores. The bottom five ranged from only 2 percent knowing who it is for and liking it, to 12 percent. Those who say you only see the best of American advertising during the Super Bowl are way off the mark. More than a few giant companies who should know better are wasting millions on the Super Bowl.

Third: What gives the most bang per buck?

How do these results compare with the number reached and affected by the average commercial aired on the networks during prime time? Here's where things get interesting. On an overall basis, 44 percent of the Super Bowl commercials reached and affected more people than the average prime-time network commercial, after allowing for differences in expenditures. So even though the overall split is close to 50/50, prime network time is a slightly better buy. But the odds change dramatically when you separate Super Bowl commercials into those that were only aired on the Super Bowl and those that received a lot of additional airings. That turns out to be one of the main differences reflected in the two sets of top commercials listed previously. The first set with the top recognition scores all had a lot of additional exposure. The amount spent to air a commercial has its biggest effect on recognition. It accounts for much less of the variation found in the impact advertising has on those who notice it. (Two years ago the measurements we were taking could account for 63 percent of the variation in overall performance. Last year we got it up to 79 percent. We are working on additional measures now to see if we can get further improvements this year.)

When we broke out the commercials that only aired on the Super Bowl we found 65 percent reached and affected more people than would have been expected if a similar amount had been spent to air the commercial on the networks during prime time. This backs up the much-discussed findings of John Philip Jones 2 that advertising's effect is immediate, and it's the first exposure that has the big effect. Airing a commercial on the Super Bowl delivers the largest possible number of first-time exposures. Spending the same amount on any other combination of programs is almost certain to deliver a substantial number of people who are seeing the commercial for the second time or the third time, etc.

As expected, the Super Bowl proves most cost efficient in reaching men and sports fans. But it is also more efficient in reaching college graduates, professionals and executives, skilled blue-collar workers and those under 30 than it is in reaching their opposite counterparts.

To get the exposure data we had Nielsen's Monitor Plus tell us when each Super Bowl commercial was aired on other programs during the survey period and how much was spent to air it. Comparisons of performance scores were made with BRC norms for prime-time network commercials that have been the subject of seven articles in the trade journals during the past 15 years 3.

Fourth: What approaches worked best?

Knowing which was most effective is one thing. Understanding why is another. That's where BRC's diagnostic measures provide some answers. Their use is illustrated in the comparative performance chart for the average food and beverage commercial on the Super Bowl. It is the category that performs the best on the Super Bowl and the chart shows why. The mood was far more important than the message, particularly the humor, uniqueness and the pace. Their messages were not seen as being nearly as persuasive, credible or clear as in the average prime-time network commercial. Yet these were very successful commercials, as shown by the above-average impact on purchasing interest. The above-average performance scores on the top half of Figure 1 show these to be successful examples of using the peripheral route to persuasion and behavior modification.

Figure 2 is an attempt to pull together everything we're learning from these studies. It shows the things our multivariate analyses have found to be most important in producing effective Super Bowl commercials, along with specific examples of executions that have worked in each area and executions that didn't work.

Notes

1 Haley & Baldinger, (1991) "The ARF Copy Research Validity Project," Journal of Advertising Research, 4-5/91.

2 John Philip Jones (1995), "Single Source Research Begins to Fulfill Its Promise," Journal of Advertising Research, 5-6/95.

3 Joel S. DuBow (1995), "Advertising Recognition and Recall By Age - Including Teens," Journal of Advertising Research, 9-10/95.

References

Biel & Bridgwater (1990), "Attributes of Likable TV Commercials," Journal of Advertising Research, 6/90.
Aaker & Stayman (1990), "Measuring Audience Perceptions of Commercials and Relating Them to Ad Impact," Journal of Advertising Research, 9/90.

Stayman, Aaker & Bruzzone (1989), "Types of Commercials Broadcast in Prime Time: 1976-1986," Journal of Advertising Research, 6/89.

Aaker & Bruzzone (1985), "Causes of Irritation In Television Advertising," Journal of Marketing, 49, 47-57.

Aaker & Norris (1982), "Characteristics of TV Commercials Perceived as Informative," Journal of Advertising Research, 4/82.

Aaker & Bruzzone (1981), "Viewer Perceptions of Prime Time Television Advertising," Journal of Advertising Research, 10/81.