Determining the return on research can be tricky but is worthwhile

By Brett Hagins | May 1, 2010

Reading time: 10 minutes

Abstract

Continuing his article series on demonstrating the value of research, Brett Hagins draws from a larger study of Quirk’s readers and in-depth interviews with client-side researchers to explore the merits and methods of quantifying the economic value of research.

Research Topics:: Executive Interviewing | Quantitative Research | The Business of Research
Industry/Market Focus:: Research Industry
Content Type: Research Methodology

Share Print

The ROI on calculating research's ROI

Editor’s note: Brett Hagins is the senior partner of Research Innovation and ROI Inc., Plano, Texas.

In partnership with Quirk’s, Research Innovation and ROI Inc. conducted a study called the Four Rs of Research: Revenue, Risk, Results and Return. It involved an online survey, 75 in-depth telephone interviews with corporate research executives and a literature synthesis that included adapting best practices from other functional areas, such as sales, to the corporate research function.

The broader study had three aims: maximizing the business impact of research; reducing unnecessary costs; and more explicitly quantifying the economic value of research. (Information about the full scope of the study is available at www.researchinnovationandroi.com.)

Most organizations do not quantify the economic value of research explicitly - mainly because it’s hard to do! But our hypothesis is that the failure to demonstrate the economic value of research has led to the commoditization of the research function in many departments where the dominant focus is on cost savings.

The benefits of quantifying the economic value of research include:

increased credibility with senior management and other functional areas;
increased focus on the factors in the processes that drive the most economic value;
a greater probability that research will be viewed as an investment rather than an expense;
more ammunition in defending the research budget;
better prioritization and allocation of research resources;
more incentive for the research department to stay involved through execution.

Through our study, we have developed 11 methods that may be used as a proxy for calculating return on investment from research. All of the methods have the following things in common:

They make an explicit connection to one or more business impacts. While understanding awareness, image, etc., is a valuable research objective, these things are not quantifiable business impacts. Quantifiable business impacts include but are not limited to such things as increased customer retention, decreased marketing costs, decreased cost of goods sold through the elimination of unnecessary features or benefits, decreased customer support calls, decreased customer acquisition cost, increased share of wallet. We would even define risk mitigation as a business impact when it is quantified financially. Intermediate benefits in learning and knowledge, which fall short of a business impact, must eventually have a business impact to calculate an ROI - although in some cases, it takes years of research to turn the boat around.

We favor a more conservative approach which only calculates the ROI associated with research that has an immediate business impact. For intermediate gains that fall short of this (diagnostic research resulting in key learning but no direct association with a business impact per se) there must be additional research and or a marketing investment at a later time to capture the benefit of that intermediate learning. We would argue than that the cost of the intermediate learning should then be allocated at the time that the business benefit is realized.

They can demonstrate that something happened as a result of the research that would not have happened otherwise and quantify the financial value of that difference OR demonstrate that risk was mitigated and quantify the financial value of the risk reduction.

They test several working hypotheses as to how the business impacts could be achieved in advance of the research, and build in mechanisms to test those hypotheses.

Thinking about the business benefits above, one of our beliefs as to why research is commoditized in some organizations is that there is too much emphasis on passive diagnostic measurement over prescriptive recommendations. Diagnostic measurement may be an initial first step. However, it frequently does not go far enough to drive the real impact. In theory, a rigorous emphasis on diagnosis leads to using that learning to drive improvements, but in practice it often serves as a rubber stamp to validate success or is methodologically dismissed if it does not validate that success. Therefore, it is critical to have the working hypotheses at hand.

In addition, you must have appropriate discussions with stakeholders to set action standards. If, on the other hand, diagnostic research truly leads to a decision to halt unprofitable marketing investments, then you can demonstrate a return on investment.

They make the projection of ROI as conservative as possible to maintain credibility. In order to do this, we recommend moving beyond differences in revenue and actually deducting the cost of goods sold, marketing investments and other incremental costs.

Note: Researchers are their own worst critics. The assumption many researchers make is that if these methods are not perfect, not 100-percent bulletproof, it is not worth making the attempt. We found during the course of the depth interviews exactly the opposite: The organizations that made the attempt were given respect even though the methodology was not perfect because they made the effort - they documented all of the assumptions, caveats and vulnerabilities of the method up front and were conservative in their calculations.

We would argue that getting as close as possible to demonstrating return on investment - even if it is not perfect - is critical in elevating the status of the research function.

Defining return on investment

Before exploring one of the methods, we should review how we are defining return on investment and other concepts.

ROI, as we are defining it, involves:

1. Determining the net margin that can be attributed to research.

2. Subtracting the cost of research bundled with any incremental marketing or operational investment associated with following research recommendations.

3. Dividing the result by the total (Research Investment + Incremental Marketing and Operations Investment).

Net margin attributable to research: Incremental revenue associated with research minus the cost of goods or services minus other costs (including marketing and administrative expenses).

Net present value: If a research ROI measure is to capture the full impact of the research investment (over a number of years) then it will have to include profits and costs from subsequent years as well as the initial year. Future cash flow is not as valuable or reliable as more immediate cash flow and so these projections must be discounted to account for this. Calculating the net present value of the research in the initial year and as well as the net present value of cost of goods sold would be one way to estimate the residual economic impact of research beyond the base period. We include an illustration of an ROI method later in this article that only looks at return for Year 1 for simplicity purposes.

Incremental marketing or operations investment: This would include any new marketing or operations investment incurred as a result of following research recommendations. It would not include the base marketing investment made anyway.

A conservative definition of ROI must account for not only incremental revenue differences resulting from following research recommendations (or risk mitigation) but deduct costs and account for incremental cost differences attributable to following research recommendations.

Of course, if cost reductions or savings are driven by the research, then these deductions are not necessary (except for any initial implementation cost necessary to realize the cost difference) and there is a more direct impact than in the case of incremental revenue differences.

An illustration: revenue maximization

In any kind of quantitative test where multiple items are tested among consumers (these can include different executions of ads, different price points, product concepts, package designs, names, etc.) it is possible to forecast the incremental revenue associated with the selection of one item over another provided that key metrics such as purchase intent, frequency, etc., are captured.

This method necessitates that research establish which concept, ad, price point, name or package design would most likely be chosen in the absence of research. We recommend that this question be included as part of the research screening process during the initial stakeholder meetings with your internal clients. At that early stage, stakeholders may not have given the matter sufficient thought to identify this, so it may be necessary to revisit which item would be chosen after the research has been designed but prior to it being executed.

Another approach that could be used if the research has already been completed and this information is not available is to take the average of all concepts, ads, names and prices tested and compare the difference between the average and the actual item chosen as a result of the research.

Keep in mind that the incremental revenue gained from the selection of one concept over another, one price point over another (name, package design, advertisement) is a starting point for ROI, not the destination. We must factor in cost of goods sold, marketing investment, differences in design or execution costs of the concept selected over the other if they are not equal. After all these expenses have been deducted, what is left could justifiably be construed as the incremental revenue attributable to the research itself.

In the course of the depth interviews conducted for our study, several who used this method indicated that the organization did not execute as planned and therefore the potential ROI was not realized. Therefore, the projected ROI calculated on the front end of a project must be factored down retroactively if the marketing investment is scaled back. The front-end calculation represents the maximum potential ROI resulting from the research and would favor not doing the research if the ROI is questionable, given all of the subsequent actions that have to occur to fully realize that ROI.

It would be easy enough to adapt this to advertising research where one was calibrating differences in purchase intent based on an advertising concept test or differences in awareness. You could also apply the same principles to a media selection calculator illustrating how efficiently different types of investments reach the target audience relative to the dollars spent.

If management does not choose a different concept than what was originally predicted (which validates that initial inclinations were correct) then it may be more appropriate to use a risk-mitigation method. In the chart, “Composite Concept Score” refers to however a quantitative measurement is derived to project trial: It could be as simple as a top two-box score for purchase intent factored for appropriate industry norms or it could be composite percentage calculated from a variety of metrics including purchase intent, uniqueness, believability, etc. This also assumes (in this case) that investments are the same regardless of which concept is implemented - although these assumptions can be easily changed. This same method could also be used to show net cost savings if a concept with negative ROI is halted due to research.

We also recognize that the above method oversimplifies the contribution of research in that it focuses purely on the incremental value driven from the selection of one concept over another and does not include (for example) qualitative research to improve the concepts themselves prior to quantitative testing.

Based on our studies, some other methods of calculating the ROI of research include:

Risk mitigation: Applicable if research does not drive a different decision but substantially reduces the risk associated with a decision.

Cost reduction: Applicable if research drives cost savings resulting from the elimination of an unnecessary feature or service element.

Cost savings (internal vs. outsourced research): Illustrate savings associated with doing research in-house versus outsourcing.

Customer retention: Quantify the revenue associated with saving high-risk customers or high-risk customer segments through intervention or process improvement.

Campaign analysis: Quantify the economic value of research used to drive the selection of one direct-mail piece over another, one e-mail over another, etc.

Opportunity and risk quantification: Quantify the value of economic value of identifying opportunities or threats in the marketplace (as for example through a tracking study or segmentation research).

Sales promotion: When research is used to support sales efforts, quantify the economic value of the inclusion of research in sales presentations or white papers as a precursor to sales calls.

Time savings (syndicated research): Quantify the financial value of time saved using pre-purchased research versus doing primary research or having executives gather needed information from scratch.

Usability impacts (Web): Quantify the economic value of usability testing for Web sites.

Research portfolio: A method for calculating ROI for the research department as a whole.

Develop additional methods

As our base of contributors expands, we will develop additional methods, including consideration for qualitative or exploratory research. Qualitative researchers especially may reject the notion of ROI. We would argue that while less exact, it is possible for departments to demonstrate the economic value of focus groups. If, for example, a focus group reveals that messaging in a particular country is mediocre or unappealing, it is reasonable to assume and illustrate that some portion of the sales in that country would not have occurred in the absence of modifying that messaging.

As long as the projections are conservative, we believe that senior management will appreciate the effort to quantify the value of research. This requires a willingness to accept the notion that an imperfect method (with heavy caveats) based on a series of assumptions is better than doing nothing. There is intrinsic value in reminding executives that a set of $30,000 focus groups uncovered a key messaging problem, the resolution to which increased sales in a market by 10 percent (or roughly $600,000).

While it may require some negotiation with internal stakeholders to reach a consensus on the relative contribution of research, we believe that the researcher who understands, quantifies and communicates the value of research within their organization is more likely to preserve and expand their budget than the researcher who does nothing because he or she cannot attribute an exact percentage.

Please contact us if you have a method you would like to share with others and we will share value with you in return. We welcome feedback from other suppliers in addition to corporate research executives and will give contributing suppliers credit for their ideas.