Is a longitudinal approach the secret to measuring an ad's ROI?

Taking the long view

Editor’s note: Jeri M. Smith is president and CEO of Communicus Inc., a Tucson, Ariz., research firm.

In these challenging economic times, CEOs are scrutinizing every expense, looking for ways to cut costs and improve the bottom line. Not surprisingly, the advertising budget is often targeted. In this environment, it is critical that the marketer be able to justify the value of advertising - to provide solid information on the impact that the advertising is having on the brand and what the result would be if the advertising budget were eliminated.

While it seems like a straightforward question - What are we getting for those millions of dollars of advertising spending? - advertising researchers have struggled for decades with how to provide satisfactory answers. Up until as late as the 1990s, many advertising agencies managed to convince their clients that it simply could not be done. There were too many other things going on in-market, advertising worked in often subtle ways, and - well, just trust us, sales would be much worse if you weren’t spending all that money on advertising.

Clearly, those arguments weren’t going to hold up forever; today’s environment requires tangible justification for the advertising investment. However, advertisers are still struggling with how to actually answer the question: “What is our advertising actually doing for the brand?”

Why is it so difficult to quantify the ROI of advertising? There are several reasons:

• Advertising is only one of a multitude of factors that affect sales; merely tracking in-market changes in brand measures during periods of advertising often provides an inaccurate (or just plain wrong!) picture of the effectiveness of advertising.
• Consumers, when asked, are usually unable to tell us whether they were impacted by advertising; this renders any straightforward survey questions on the impact of advertising nearly useless.
• To further confuse the issue, consumers are more likely to engage with advertising for products that they already buy - causing a serious chicken-or-egg problem with any analysis that attempts to deduce advertising impact by comparing - after the fact - the behavior of those who’ve seen advertising with those who have not.

The traditional tracking study - in which brand measures such as awareness and perceptions are tracked over time among matched samples of the target audience - has proven to be a relatively insensitive tool for isolating the effect of advertising on a brand. Often, these studies find little or no movement in brand metrics on a pre/post-advertising basis, leading the advertiser to wonder if advertising really is a total waste of money.

Continuous tracking studies, which attempt to correlate periods of high advertising pressure with changes in brand awareness and perceptions, appear on the face to provide more sensitivity to advertising’s impact. However, in practice, this type of analysis can - at best - usually only identify positive relationships between advertising spending and brand awareness. Rarely is this correlational approach able to detect relationships between advertising and measures that are actually closer to sales - brand purchasing, future purchase intentions or even brand perceptions.

Longitudinal design

In our experience, the best way to actually isolate the in-market impact of advertising is through the use of the longitudinal design. The longitudinal, or panel, design involves interviewing the same people at two points in time - typically before a new campaign (or series of executions) is launched, and again later, after the campaign has been in-market for some period of time. Isolating the impact of the advertising on a brand entails comparing changes in brand measures (e.g., awareness, perceptions, purchasing/usage) among those who have seen brand advertising during the period versus those who have not.

In Figure 1, reported brand purchasing was relatively unchanged on an overall basis before and after the ad campaign had run for a six-month period [T]. A tracking study, which measures pre/post change, would have concluded that the advertising wasn’t working. The advertiser would have been left to wonder what to do:

• Should the budget be cut, based on apparent lack of return on investment?
• Should the campaign be replaced with a different one that might be more effective?

With the longitudinal design, the advertiser would have learned that the advertising was indeed quite effective among those who actually saw and engaged with the campaign in-market; among this group [A], brand purchasing was up significantly versus the downward trends in purchasing that were occurring among those without advertising awareness [NA]. This campaign was not a candidate for elimination or replacement - it was a candidate for a greater investment, in order to expand on the positive effects that it was having.

To effectively isolate the impact of an advertising campaign using the longitudinal design, it is first necessary to determine who has seen the advertising and who has not. Our approach to this challenge has been to adopt, adapt and refine techniques first developed based on the PARM studies conducted by the Advertising Research Foundation a number of years ago. Today, we are able to determine, with accuracy, which specific ad executions each respondent in our research has seen/heard and stored in their long term memories; these measurement methods can be equally accurate for TV, radio, online advertising and the static visual media such as magazine, newspaper, FSIs, point of sale displays and outdoor ads.

Once we can categorize each respondent in our panel according to what advertising they have seen, the longitudinal design analysis can provide definitive measures of the actual in-market impact of advertising.

In contrast to traditional or continuous tracking, the longitudinal design:

1. Provides information on the impact of not advertising.

Because the longitudinal method tracks changes among people who saw advertising and people who didn’t, it provides a means of answering the question: What would have happened if we hadn’t advertised?

While conventional wisdom holds that the health of a brand will begin to deteriorate without advertising, the only way to test this theory using traditional methods is to conduct test/control market studies in which advertising is reduced or withheld from selected markets.

With the longitudinal design, this is unnecessary. As in the above example, analysis of changes in brand metrics among those who have not seen advertising over a period of time provides a clear answer to this question, and also serves as the baseline against which advertising impact is judged. Insights gained from this method are valuable in justifying continued advertising investment, based on calculating the ROI of advertising versus what would have happened without advertising.

In fact, there are many cases in which brands that are experiencing little to no sales growth actually have effective advertising; in these cases, to withhold advertising would be severely detrimental.

2. Controls for other marketplace dynamics.

The longitudinal analytic method also provides an internal control for seasonality, severe weather patterns and economic conditions that affect category/brand sales, and so on, since the baseline measures of change among those who did not see advertising reflect these marketplace dynamics.

3. Controls for selective perception.

Most advertising is subject to selective perception - that is, those who already prefer or otherwise are more highly engaged with a brand will be more likely to engage with the advertising for that brand. This fact renders post-advertising comparisons of brand metrics among those who saw ads versus those who didn’t useless. If 50 percent of those who saw advertising prefer the brand versus 25 percent who didn’t see the advertising is this because the advertising doubled preference or because those who prefer the brand were more likely to engage with the advertising than those who don’t prefer the brand?

The only way to control for this is to measure brand attitudes among a sample of the target audience before advertising exposure occurs and then go back to the same people to see how advertising has changed them.

4. Isolates the impact of each medium within a multimedia campaign.

The longitudinal design, when implemented in conjunction with ad awareness/engagement measures, enables analysis of the changes in brand metrics that were produced by each medium and media combination within a multimedia campaign. Because we can determine which sample members have seen each medium - alone and in combination with other media - we can compare changes in perceptions and behavior based on, for example, having seen TV commercials versus magazine ads versus online advertising (see Figure 2).

Not only is this level of analysis useful in identifying the contributions of each medium to campaign impact, but it can also play a critical role in guiding media reallocation strategies that result in an improvement in the overall return on investment of the campaign.

For example, an advertiser might find that it costs three times as much to achieve advertising awareness within the context of a particular campaign by using TV than by running magazine ads. However, if the TV is four times as persuasive as the print among those who have seen it, TV generates better ROI than print. Other factors that come into play, and that are part of the longitudinal multimedia analysis, include the extent to which each medium achieves impact among those not reached by the other media, synergies across media, etc.

5. Isolates the impact of non-traditional media.

Using the longitudinal design, measures can be developed that isolate the impact of non-traditional media within the context of integrated/360° campaigns. Non-traditional elements whose contributions can be quantified with the longitudinal design include online ads, Web sites, YouTube videos, product placements, advertorials, point-of-sale displays/events, sponsorships and more.

The performance of each campaign element is evaluated in terms of: engagement achieved; brand linkage; communications of brand messages; persuasive impact; synergistic interactions with other communications elements; and cost efficiency in relation to other communications elements.

Using this approach, advertisers can understand how all elements within a complex communications program are working.

6. Diagnoses effectiveness of message communications.

Thus far, the discussion has focused on the ability of the longitudinal design to isolate the impact of advertising campaigns on consumer behaviors such as brand purchasing. The longitudinal analysis can also provide insights into the extent to which the advertising is producing an impact on consumer perceptions of the brand.

For example, an advertiser may find that a new breakfast cereal campaign targeting moms is successfully building perceptions on “provides nutrition I can believe in.” However, if consumer behavior is not also being impacted, the advertiser may need to take a closer look on the motivational value of this message.

In another case, an advertiser might find that the new TV campaign is successfully building brand affinity (“for people like me”), but the accompanying radio commercials are not. Then, the task is to identify the reasons for this difference, and to build more personal affinity into the radio executions.

Predict strong performance

Advertisers frequently copy-test individual commercials and ads prior to in-market launch in order to identify those that will produce strong performance. Of course, the assumption is that the copy-test results will actually predict in-market performance.

Since our in-market research design starts by measuring the awareness, brand linkage and communications achieved by each ad execution within a multimedia campaign, it can generate insights on the extent to which the outcomes predicted by copy tests actually come to pass in-market.

For TV commercials, we have reviewed approximately 85 individual TV executions that were pretested using one of the major copy testing systems, then run (without revisions) in-market and subjected to the in-market longitudinal design advertising evaluation system.

Based on these data, the copy testing system that has produced the strongest predictive accuracy is the clutter recall method, as administered by Ameritest. Among the Ameritest cases subjected to this analysis, the predictive accuracy was 95 percent, with the system providing correct diagnostics (e.g., identifying strengths/weaknesses on specific performance dimensions such as attention and branding) in all of these cases (Figure 3).

The next-closest method in predictive accuracy was the day-after-recall method, which picked the winners/losers with 80 percent accuracy. However, in approximately half of these cases, this method failed to provide the specific diagnostic information that would have been helpful in identifying opportunities for improvement.

The Ameritest print testing method has also achieved predictive accuracy based on correlations with our actual in-market performance measures. (We do not, however, have sufficient print copy testing data from other providers to report on the accuracy of other systems.)

Of course, other variables do come into play in-market. For example, a commercial that airs within the context of a complete, multimedia campaign may perform better in-market than one that is introduced in isolation, without other, synergistic communications elements.

On this basis, we encourage advertisers who desire the best ROI for their advertising to:

• Copy test their new creative, using a system that is both as predictive as possible and that provides accurate diagnostic feedback (to generate insights into how to improve the tested, and future, executions).
• Utilize a longitudinal in-market advertising evaluation system, to identify how the entire campaign and the individual campaign elements are contributing to brand behavior and perceptions in-market.
• Take advantage of the learning environment produced by this combination of advertising research systems to continually get smarter about how advertising works for your brand.

Number of factors

A properly designed and implemented longitudinal design study must take into consideration a number of factors, including panel size and selection, design factors to insure against sensitizing respondents in the pre-wave interview, timing between phases and the special analytic methods required for proper interpretation of longitudinal data.

However, with the right practices in place, the longitudinal design can provide advertisers with better answers to the question: “What is my advertising campaign doing for my brand?”

The longitudinal design, by tracking the changes that occur among those who have seen your advertising versus those who have not, insures that the research is sensitive to the changes produced by advertising, while not mis-attributing to advertising changes that were actually caused by other factors.

The results of a longitudinal design study identify the contribution of each medium within a multimedia or fully-integrated 360° campaign, and help the advertiser to optimize the ad budget across media. On this basis, the longitudinal design can provide both a means of calculating the ROI of advertising and in-depth information to optimize future campaigns.