Learn from the experts
Editor’s note: Charles H. Ptacek is president of Charles, Charles & Associates Inc., a Gold Canyon, Ariz., research firm.
Product-market analyses and forecasts are vital to the preparation of business plans as well as the allocation of company resources. It is frequently possible and often preferable to obtain product-market information through specifically focused exploratory research methods as opposed to the more costly and time-consuming quantitative, large-sample survey methods that dominate marketing research practice. For finite universe technical and industrial markets, surveys based on random samples can produce non-representative information because high-demand concentration spells a high degree of information concentration among a very limited number of sources.
This concentration of relevant information requires that careful attention be given to the sample selection process, using representative judgment samples rather than random sampling methods. Furthermore, given the limited number of viable information sources and the level of distinct expertise, unique knowledge and focused market intelligence associated with these sources, it is often necessary to employ survey methods that bring together the opinions and judgments of experts. Product-market analyses and forecasts are often based on expert judgments, especially for relatively new or rapidly changing markets.
Interactive forecasting method
The Delphi technique offers a useful way to implement many of the basic principles for expert market analyses and forecasting. The Delphi technique is a systematic, interactive forecasting method based on independent inputs of selected experts or knowledgeable persons. The Delphi method was developed over a period of years by the Rand Corporation at the beginning of the Cold War to forecast the impact of technology on warfare. Over the past five decades, this procedure has become a popular tool in long-term sales forecasting, technological forecasting, business strategy analysis and new business development.
The Delphi research paradigm is one of the most important exploratory techniques for investigating future manifestations in finite technical and industrial universes. For example, the iterative aspect of the method, as well as the in-depth normative nature of the investigative paradigm, lends itself to a wide range of technical and industrial new product development situations.
In instances where the market for a new product is not well defined and/or the product concept is emerging, the technology is still fluid and history seems inappropriate for forecasting future events, the Delphi technique can produce reliable and valid market intelligence. Rich in diagnostics, applications employing this method have demonstrated that the level of information derived from this approach can be used to direct development efforts as well as provide insightful marketing programs.
One-on-one interviews
The Delphi interview method represents an in-depth iterative research effort involving one-on-one personal and/or executive telephone interviews with a representative sample of experts or knowledgeable persons from selected target market businesses. The two keys to the successful use of experts with Delphi are 1) identifying who they are, and 2) achieving a convergence of opinion.
The selection of the panel of participants is critical to the success of the study. The pool of experts is often identified via the key informant technique. The key informant technique represents a form of networking with knowledgeable persons in an industry, thereby identifying which persons are considered most knowledgeable; these are the persons who are recruited as experts. The pool of experts is initially identified during the process of collecting secondary information from trade publications and associations as well as industry source referrals. Generally, less than 100 experts are recruited as participants for a Delphi study. Studies with more than 100 experts make administration of the Delphi study overly complex.
The Delphi method relies on repeated measurement and controlled feedback to obtain the consensus opinion of a group of experts. Opinions of experts are converged into an informed consensus through a structured multi-step recruiting and polling process administered by a skilled facilitator. Qualified participants are recruited and interviewed using executive interviewing procedures.
Based on previous experience, interviews are typically over one hour, depending on whether the respondent is providing critical baseline information or supportive data. The initial evaluation process includes the analysis of information needed to provide preliminary assessments. This information is analyzed with respect to the participant’s area of expertise as well as how this might bias certain estimates, the completeness of the data obtained and the exogenous and endogenous variables affecting this information.
Participant evaluations that deviate from the general consensus are analyzed and substantiated reasons for variance are cited. When unsubstantiated evaluations are identified, participants are recontacted to clarify their judgment. These participants are provided additional information obtained from other contributors and given the opportunity to revise their judgment.
Hence, in addition to the original in-depth Delphi interviews, recontact and supplemental interviews are usually performed. Recontact interviews are conducted to achieve a convergence of opinion on critical issues or to obtain additional information. Supplemental interviews are usually required when multiple sources exist and these particular experts are crucial to the convergence process. Past experience indicates that supplemental and recontact interviews provide a level of validity that must be achieved in finite-universe Delphi studies.
All expert interviews are taped or digitally recorded and transcribed for analysis. The in-depth qualitative analysis can best be described as a psychological content analysis, where the emphasis is on creative analysis, synthesis and interpretations of participant discussions. Quantitative analyses are focused on obtaining the most reliable consensus of opinion from a group of experts. This includes performing follow-up interviews interspersed with controlled feedback.
With this Delphi approach, human intelligence, rather than a mathematical model, is the primary provider of facts, knowledge and information. Experts converge on the “best” intelligence by the application of mental processes rather than by the use of formulas. However, it is important to note that analysis should be carried out using metric and/or non-parametric statistical procedures in order to identify appropriate convergences and divergences in responses.
A case study
In 2000, U.S. book sales amounted to an estimated $24.5 billion dollars. Although positive demand for books is likely to continue over the next 10 years, more uncertainty exists for book publishers, manufacturers and their suppliers. Breakthroughs in digital technologies have begun to impact printing and publishing processes and alter the roles and relationships of major industry players. Additionally, emerging electronic technologies have created new media for delivering intellectual information that has the potential to compete with traditionally printed books.
The overall purpose of this study was to provide printers and publishers in America with insight into forces influencing future book demand, book publishing and printing processes and relationships between major players in key book industry segments. This assignment was intended to explore and define how book publishing and printing is likely to evolve in the coming decade. The effort included forecasting the growth of key technologies and assessing the impact of on-demand printed books as well as providing demand forecasts for key book segments.
This Delphi application started with a Stage I exploratory investigation incorporating secondary information and in-depth interviews with a few selected industry experts. The general objective developed for Stage I was to investigate future book publishing trends from the perspective of major industry participants and to provide a historical timeline documenting major technological and societal milestones which have impacted the book publishing industry over the past 20 years.
We were interested in understanding what had happened to the market for books during this period and, based on that understanding, developing an informed judgment as to what could happen in the years ahead for the purpose of developing a set of futuristic scenarios. In addition, Stage I provided a baseline forecast of book sales, along with estimates of how new technologies and other factors including demographic changes are likely to affect the baseline forecast.
Stage II provided an in-depth iterative Delphi survey with leading book publishers, printers, suppliers and other technology experts. Both the publishing and book manufacturing industry sectors are dominated by a finite group of conglomerate companies, where the top 10 publishers account for more than 80 percent of the annual book revenues, and the top 10 book manufacturers account for 50 percent of the annual book manufacturing revenues.
In total, 21 out of 25 of the largest publishers were recruited. Seven out of 10 of the largest printers were recruited and interviewed, as well as leading book industry suppliers, book distributors and retailers. In total, approximately 100 knowledgeable individuals within these companies were asked to evaluate the baseline forecast derived in Stage I and provide their expert opinion on factors influencing book demand. These opinions were used to adjust and refine the Stage I baseline forecast and to project the demand for books and related supplies over the next decade using scenario forecasting procedures.
Accuracy of forecasts
One indicator of the value of Delphi as a forecasting tool is its accuracy, and several technical reviews have provided evidence on the accuracy of Delphi forecasts. The overall conclusion is that forecasts from Delphi groups are substantially more accurate than forecasts from unaided judgment and traditional groups (e.g., focus groups). Market forecasts, however, often represent a very small portion of the total value provided by a diagnostically rich Delphi product-market analysis and forecast. For example, in addition to the current market size estimates and long-term demand forecasts, the book market Delphi study provided a thorough profile of all key book market segments and associated trends influencing demand.
Table 1 presents year 2000 book industry forecasts for four key book sectors - trade, religious, education and professional - as well as a total industry forecast. Year 2000 forecasts exist for annual sales dollars and number of units sold for each book sector. The forecast estimates were excerpted from a 1995 “Books in the 21st Century” Delphi study providing market forecasts for the period 1994 to year 2000. The observed or reported book sector numbers were excerpted from a 2003 repeat Delphi study, “Book Market Outlook 2003-2012.” The Delphi methodology and forecast procedures employed in both studies were exactly the same and were performed by the same research consulting company. Both book market Delphi studies were sponsored by Print Industries Market Information and Research organization.
As listed in Table 1, annual dollar forecasts for the total book industry are within 1 percent of the observed/reported $24.7 billion for year 2000. Annual dollar sales forecasts for the religious and professional book industry sectors are within 5 percent of the observed/reported dollars for year 2000 while forecasts for the trade and education book sectors deviate more than 10 percent above the observed/reported dollars for year 2000. Correspondingly, year 2000 forecasts for annual units sold are approximately 13 percent higher than the observed/reported units sold. Most of this deviation (78 percent) can be attributed to the education book sector, where over a seven-year period (1994-2000) astronomical price increases contributed to the significant underestimating of annual sales dollars and overestimating of annual books sold for the year 2000. Although the price increase trend was identified during the Delphi investigation, experts severely underestimated the extent of this trend for education books, especially college textbooks. Overall, however, forecasts for three out of four book sectors as well as the total industry forecasts represent reasonably accurate estimates for a seven-year period.
Exploit specialized knowledge
The purpose of the Delphi methodology is to exploit the specialized knowledge and judgment of experts to ascertain critical information that can be used by management to make informed decisions. The use of judgmental input (expert evaluations) in decision-making is hardly avoidable today and is rapidly becoming the foundation of many of the more advanced and sophisticated decision models in marketing. At least part of the Delphi ’s appeal is attributable to the flexibility of the methodology. The process is adaptable to a wide range of investigative applications; however, in the corporate business environment, Delphi is primarily used to facilitate the formation of futuristic group assessments.
The approach used with Delphis shares the common characteristics of 1) offering feedback in the form of individual contributions; 2) providing an assessment of group opinion while minimizing the more undesirable aspects of group interaction; and 3) providing interactive convergence by allowing individuals to revise their original responses following the initial assessment of group opinion.
All of these characteristics differentiate the Delphi research paradigm from traditional quantitative survey research methods as well as traditional qualitative focus group methodologies. The Delphi paradigm meets five key guideposts of sound inquiry: objectivity, reliability, validity, intensive analysis and marketing applicability. We have used the technique to provide accurate estimates of future circumstances and demand-driven technological forecasts, as well as future estimates of new product market potential.