Editor’s note: Brett Hagins is the senior partner of Research Innovation and ROI Inc. Plano, Texas. Additional information of the study mentioned in this article is available at www.researchinnovationandroi.com.

My firm, Research Innovation and ROI Inc., has completed the first edition of annual study called the Four Rs of Research: Revenue, Risk, Results and Return. The objectives of the study are to help client-side researchers maximize the business impacts associated with research initiatives, reduce unnecessary costs and more explicitly quantify the economic value of research while linking it to financial outcomes.

The basis for the study was a quantitative benchmarking survey focused on best practices (designed collaboratively with input from sponsors), followed by in-depth telephone interviews and a literature synthesis that involved adapting best practices from other roles - such as sales - to problems commonly faced by corporate researchers.

For the first phase, Quirk’s invited its database of client-side researchers to participate in our online benchmarking survey. The survey invitation included additional screening to verify client-side status and some level of budget responsibility. Almost 350 client-side researchers responded to the survey - with representation from both larger and smaller companies; most industry groupings; and different departmental sizes, reporting structures and budgets.

My firm then conducted 75 in-depth telephone interviews among client-side research executives (research managers/directors, vice presidents and marketing vice presidents) to study successful projects and best practices among client-side researchers. Information about the full scope of the study is available at www.researchinnovationandroi.com.

This is the first in a series of articles that will discuss the results of the study. Over the next several months we will cover topics such as: quantifying the economic value of research; encouraging the research department to take more risk through the use of performance evaluation criteria; overcoming barriers to research implementation; and best practices in reducing research costs.

Not actionable enough

The focus of this article is on enhancing return on investment from tracking studies. While suppliers love tracking studies as a recurring source of revenue, my view is that they frequently are not actionable enough and that well-planned custom research can yield more useful data.

In our benchmarking survey, we asked corporate research executives which research methodologies they personally have been involved with in the last 24 months and which of these (if any) drove strong business impact for their respective organizations. The ratios of involvement to impact are shown in Figure 1 for various research methods and/or study types.

Part of our depth interviews focused on identifying best-in-class questions that senior researchers use to get information during initial meetings with their internal clients.

A sample of these questions, corresponding with a possible business application of tracking studies, are shown below:

“For this tracking study, what business objectives (highlighted in bold) do you hope to achieve?”

Diagnose marketing effectiveness

“How will you judge success? What are the goals of the marketing campaign(s)? Which metrics are critical for you - awareness? Preference?”

“What kinds of movement in these metrics would you expect to see if the campaign is successful? Three points? Five points?”

“Where will most of our marketing efforts be concentrated for next year (markets, developing consumer or firmographic profiles, etc.)?”

“What is your marketing budget for next year? The impact of a marketing campaign of that size may not register in an ongoing tracker - let me suggest some other options…”

“Would it be helpful to get a clear read on the ad, etc., so that we can measure the impact among those exposed to it?”

“Do we know what the media plan is? Which markets will be the focus? What will be the weight of our media in each of these markets?”

“Can you share the marketing calendar showing when the events will occur?”

“How much time do we have to get a baseline measure before the first marketing event?”

“Do we know what competitors have planned as well or what their relative spend is?”

“If our share-of-voice is low, would you consider being able to hold metrics steady or preventing a decline one measure of success?”

“In addition to yourself, who else will have responsibility for the impact of the campaign? Who else do I need to talk with?”

Serve as an early warning system for threats to the company

“What specific potential threats are you worried about?”

“What trends do you see in our industry, either positive or negative, that may impact our relationship with customers, now or within the next few years?”

“What might be a leading indicator that the concern may come to pass?”

“Which existing competitors pose the biggest threat to us? Which emerging competitors are gaining momentum quickly even if they may be small now?”

“Outside of our industry, what alternative/substitute products or business models could potentially impact our business?”

“What percent of the market would have to validate this threat before you took action?”

“Which segments of the market are most vulnerable to the threat?”

“What are some potential actions you might take to mitigate the threat?”

“What are the sources of internal opposition to these actions right now?”

Serve as an ongoing monitor of brand and reputation

“Into which new areas are we trying to extend our brand?”

“What are some factors that might cause our brand to decline or lose relevance in the marketplace?”

“Are there any plans to change our logo, positioning, etc?”

“Aside from our core brand, are there specific products and services (or sub-brands) that we need visibility on?”

“My understanding is that our current brand positioning is x, y and z - Is that right? What have I left out?”

“I can think of 10 touchpoints where customers are exposed to our brand right now? Which of these are the most critical?”

“Here’s a list of 20 different measures that we could assess on an ongoing basis - which of these are most critical? How will you use that information?”

Assess the impact of new organizational programs and key changes

“What organizational changes have we made in Area X over the last 12 months?”

“Would the market have any visibility on this? Would our customers? What segments of customers, if any, would be most likely to notice the change?”

“Tell me about key operational changes you have planned in [insert customer touchpoint, area of responsibility] for the coming year.” [Repeat for each touchpoint related to the brand.]

“What has been the anecdotal feedback from customers thus far about Program X? What additional hypothesis do you have as to what we might find out?”

Identify opportunities in the marketplace

“What hypotheses do you have about opportunities out there that exist that, if validated, you could take action on?”

“What percentage of the market would have to validate the opportunity?”

“What form might that validation take in the context of this type of study?”

“Which segments of the market do you believe have the greatest growth opportunity?”

“From a business perspective, which of these do you currently see as our biggest strategy to drive growth?” [Probe key stakeholders for opportunity hypotheses based on each objective.]

•   Increasing share-of-wallet from existing customers

•   Get new customers

•   Sell more products to existing customers

•   Sell existing products in new markets

•   Pursue growth from different channels - distributors, the Web, etc.

•   Sell new products in new market segments

•   Reduce unnecessary costs

Screen it out

The value that a tracking study could provide has to be weighed against the value that other more custom approaches could provide for the same money and the same investment of time and energy. For each potential reason to do a tracking study there is a corresponding reason to screen it out or use a different approach.

Diagnosing marketing effectiveness. Tracking studies can be ineffective for this unless the company’s marketing efforts are on a massive scale. It is like firing a water gun into an ocean and then trying to see a ripple from an airplane at 30,000 feet. It requires a great deal of customization in the design to do this effectively. This is often better left to a custom study focused on particular campaigns, events, etc.

Monitoring brand and reputation. Unless a client is in a really fast-moving, high-growth industry, brand and reputation do not change much (generally) over the short term. A director of research once told me that his firm had to keep its tracking study going in case “bad news” came out about the company that adversely affected its reputation. Shortly after this prophetic conversation, the company was indeed embroiled in a scandal in the media, which was reflected in some of the results. The study did not provide any guidance on what to do about it, however, and the entire marketing research budget was eliminated a few months later.

Although tracking studies may monitor brand and reputation (and occasionally diagnose the impact of a perception-changing event accurately), they are still too focused on the diagnostic (what happened) and not on the prescriptive (how to improve things).

Serving as an early-warning system for threats or to identify opportunities. Unless they are designed with forward-looking questions that anticipate changes, most tracking studies validate what others in the organization already suspect. They are a lagging indicator, not a leading indicator. It is possible to design a study however that predicts change and does not simply validate it. But it is critical to go through a process of envisioning alternative futures first so that the organization will not simply be in denial when it sees a threat or dismiss a potential opportunity because it does not confirm preconceived assumptions.

Diagnosing the impact of operational changes. Tracking studies are frequently not targeted narrowly enough to do this.

For all these reasons, clients are often better off doing more custom, targeted research projects that are more focused. While a tracking study can be used for a lot of things it often is not particularly effective at any one thing.

At a minimum, work with suppliers who realize the limitations of tracking studies, as they will likely be more inclined to suggest alternative solutions.

Not tracking the right things

One of the reasons why tracking studies are ineffective as an early-warning indicator of competitive threats (or opportunities) is that companies often times are not tracking the right things. Their view of what could happen in the future is heavily shaped by the politics of the day and the existing social order. If a tracking study does spot a threat or an opportunity, the organization may be in denial if it does not fit with preconceived assumptions.

Most organizations subscribe to an official view of the future pushed down by senior management and do not bother to consider alternatives. Some companies have effectively used a process called scenario planning to develop alternative views of the future. They look out on the horizon - five years, 10 years or whatever time frame is most appropriate for the company - and then develop stories about the way their industry, markets and the world might look like at that future point. They then figure out what the leading indicators of each vision are and incorporate questions into their tracking study to help identify these indicators. This involves a process of doing in-depth interviews and including industry experts from the outside - even experts from indirect competitors who fulfill the same core need in other ways. Typically, a positive, neutral and negative scenario is developed based on competitive, environmental, economic and technological trends. They then flesh out fully-descriptive story lines for each scenario, identify leading indicators of which one is coming to pass and determine the best course of action given different circumstances.

Organizations unwilling to challenge conventional wisdom and the assumptions they make will tend to discount early-warning signals. As late as 2003 for example, I did a small project for a newspaper that did not want to track the Internet as a source of news and information. Blockbuster had many tracking studies in place, but it did not respond quickly to Netflix until its impact did real damage to Blockbuster’s market share. Unless an organization has thought about possible opportunities and threats in advance, and thought about the implications of those opportunities and the most appropriate path for each situation, it is very hard to break out of the business-as-usual mode.

Merge with other forms

Tracking studies should merge with other forms of research to provide greater depth. For example, intercepting respondents in the middle or at the conclusion of a study for an interactive chat session (such as through iModerate, which I have used) can provide more depth on targeted subgroups of interest.

For instance:

•   intercept those who have awareness of company or competitive advertising to probe for recall and effectiveness;

•   intercept lead users or early adopters to probe on new developments in an industry that may not have emerged yet among the entire respondent base;

•   intercept respondents when particular events or conditions occur: declining share-of-wallet, growing share-of-wallet, etc.

If a company is tracking the same set of respondents longitudinally, they may wish to have its suppliers give those respondents Webcams to facilitate subsequent depth interviews online and help bring the research to life on the back-end. This allows stakeholders to make more of an emotional connection with the data, which is important in helping them internalize it.

Finally, consider having your supplier recruit a select group of industry experts and show them a portion of the results to solicit their opinions on interpretation, implications and what each company should be doing to accelerate strengths and mitigate weaknesses.

Obtain competitive bids

A huge portion of any market-level tracking study is tied up in the sample costs. Suppliers should obtain competitive bids from at least five different sample providers and negotiate with all of them to get the best possible price. In the current economic environment, you should expect quality and good value. Unfortunately, many full-service agencies don’t make this a priority and then turn around and mark prices up further. Corporate research clients who purchase a lot of outside sample should develop direct relationships with sample providers or with a broker who specializes in this area. 

Easy to misinterpret

It can be easy to misinterpret findings from a tracking study. Early in my career, I noticed that after a large advertising campaign, my company had increased awareness but had lower favorability scores. Did the campaign cause our favorability scores to go down? I finally realized that the new people who became aware of the company as a result of the campaign had lower favorability scores because their awareness was more superficial. So comparing the scores on that metric (pre- vs. post–campaign) was like looking at apples vs. oranges.

One of the challenges of interpreting tracking studies is that the sample size is often insufficient to draw conclusions. Leading companies will go back to the same set of respondents longitudinally at certain intervals because a paired sample is more powerful than an unpaired sample in detecting what has changed. With the appropriate experimental design, it can also be more effective in isolating the net contribution of a company’s marketing efforts from other factors, such as changes in the economy, share-of-voice, etc. Going back to the same set of respondents from a previous wave obviously will distort some metrics like awareness, and therefore we recommend using a combination of fresh and previously-surveyed respondents. Sample companies with strong panel retention are best for this longitudinal approach.

Automating their reporting

By now, every research firm should be automating their reporting for tracking studies, which virtually eliminates human error, accelerates delivery time and allows more time for value-added insights. When there is quality reporting automation software available for $10,000, it is difficult to understand why even client-side researchers who do this work in house have not already made this investment.

In addition, consider reporting of tracking studies that focuses on relevant changes and key insights. No one needs to see 50 slides showing them that metrics are holding steady. Electronic dashboards that highlight changes in key metrics are more useful, and some leading programming tools have started integrating these into their offerings. Qualtrics was mentioned twice on an unaided basis during the depth interviews as a firm that has made this kind of investment of behalf of its clients. Also, one-page summary memos that are customized to particular audiences are more effective than a 50-page PowerPoint deck few will read. When most data is already available in charted form in online portals, consider having suppliers focus on executive summaries - potentially increasing usefulness while simultaneously lowering cost.

Quantify the financial size

One of the ways to make senior management care about opportunities and threats highlighted in tracking studies is to quantify their financial size. Slowly-emerging threats may require projections that look several years ahead based on existing trends for executives to fully appreciate their significance. Find ways to link softer metrics with harder ones. One leading company we talked to had quantified that every 10-point increase in awareness translated into a one-percent increase in market share.

In addition, an internal stakeholder survey conducted at the outset that documents views from all key players is useful in demonstrating that the research effectively changed perceptions and drove some action that may not have occurred otherwise.