The confluence of data mining and market research for smarter CRM

By Kenneth Elliott, Richard Scionti, Mike Page | May 1, 2004

Reading time: 13 minutes

Abstract

For companies with large numbers of customers, data mining and research are often used to investigate customer behavior and attitudes. The authors argue that the two must be integrated to create a truly holistic view of customer behavior and present a blueprint for how to do so.

Research Topics:: Customer Loyalty | Customer Satisfaction Studies | Data Analysis | Data Processing | Database Development/M.I.S.
Content Type: Research Methodology

Share Print

Two rivers

Editor’s note: Kenneth Elliott and Mike Page are partners in and co-founders of Cognicient, an Elmhurst, Ill., research firm. Richard Scionti is vice president, survey and market research services, at SPSS Inc., Chicago. Further information on the research on which this article is based can be found at www.cognicient.com.

In most companies, the realms of customer behavior analysis and customer attitude analysis are worlds apart. They are like two swift flowing rivers that never meet. Behavioral analysis is typically the domain of business intelligence: tightly managed by IT and heavily focused on operational systems, data management, report servers, on-line analytical processing (OLAP) cube administration and data mining. While attitudinal analysis is the world of market research: owned by marketing, often outsourced to a market research agency, resulting in tabular reports and executive briefing documents.

However, true holistic customer analysis demands that these worlds come together. Customers both think and act. An understanding of how customers think can help explain and predict customer behavior. Conversely, customer behaviors can help explain and predict customer attitudes. Ideally, behaviors and attitudes would be analyzed simultaneously for deeper customer understanding.

For companies with large numbers of customers, data mining and market research are often employed to gain intelligence into customer behavior and attitudes respectively. Therefore, truly holistic customer analysis requires that these two disciplines be integrated. The rivers must converge. In this article, we will examine the issues surrounding the convergence of data mining and market research for deeper customer understanding.

Are data mining and market research integrated within your company?
Are you optimizing your investment in behavioral and attitudinal data for a complete picture of your customers’ intentions and actions?
Are you aware of the potential costs associated with redundant use of two disciplines to examine the same research question?
Can you create a more efficient and accurate research operation by coordinating these disciplines for deeper customer understanding?
What potential barriers will you face by trying to create a coordinated research operation?

What is data mining?

Let’s begin our examination of the convergence of data mining and market research by exploring the basic principles and common uses of data mining for customer understanding today. (We will use the term “customer” to represent the identifiable customers that are known to an organization as well as anonymous “consumers” of the organization’s goods and services.)

There are several definitions of data mining in use today. Broad definitions suggest that data mining is the exploration and analysis of large data sets. Under such definitions reporting, graphing, traditional statistics and sophisticated machine learning are all considered data mining. In this document we use a more narrow definition of data mining that stresses the discovery aspect of the discipline. Specifically, we see data mining as the iterative process of using pattern discovery algorithms to find useful and previously unknown trends and relationships in large volumes of data. These patterns help explain past events as well as predict future events.

Data mining is used in many industries where there is a need to find patterns in vast amounts of data. Perhaps its most widely recognized use is in the commercial market. Today’s businesses are using data mining to identify patterns in customers’ buying behavior; identify profitable customer segments; increase marketing return rates; prevent loss of valuable customers; estimate credit risk; identify fraudulent activity and much more.

The strength of data mining is in its ability to quickly sift through vast amounts of data to find patterns that are hidden and would otherwise be impossible to find. Data mining often uncovers unexpected patterns, which fosters new learning and insight.

According to a 2002 report from IDC, the data mining market is expected to grow at a CAGR of 13 percent to reach $823 million in 2006. This growth can be attributed to at least four key factors.

1. There are more information sources available today than ever and the amount of information is growing exponentially.

2. The explosive growth in the capacity of databases along with the shrinking cost of data storage has made it possible to acquire, store and manage more data than ever.

3. Using data mining techniques used to require complex programming skills. Today, there are powerful data mining tools on the market that are easy to use, making data mining more accessible to a broader audience. Many operational suite vendors are beginning to embed data mining into their applications.

4. The highly competitive market environment and growing customer options make customer intelligence more critical for business performance. This has created an increased appetite for rapidly finding knowledge from vast amounts of data.

Support decision making

Within the context of customer intelligence, data mining and market research are often used to support decision making in the areas of customer acquisition, customer segmentation, customer retention and cross-selling. These applications are part of the field called analytical customer relationship management (A-CRM). As described below, the insights gained from these initiatives help organizations better manage their customer interactions, improve the level of customer service, and create richer, longer-lasting customer relationships.

Customer segmentation. Under-standing customer segments is critical to any customer-focused organization. Market research derives customer segments through surveys and demographic research. Data mining uses clustering techniques to find naturally occurring groups within the customer database. While each approach individually provides insight into basic customer groups, combining these approaches yields deeper insight still. A simple illustration of this can be seen in the table, which shows variances between purchased demographic segments and clusters that are derived by behavioral, transactional and individual characteristics. Segment 1 seems to include two distinct behavioral clusters. An understanding of Clusters 1 and 2 may suggest varied marketing strategies within Segment 1. Segment 2 and Cluster 2 seem to validate each other. Clusters 1 and 3 contain two different demographics. While these two groups seem to behave the same, demographics may provide insight into differing intentions. Combining data mining and market research techniques for customer segmentation can lead to refinement of segmentation strategies and to more accurate customer understanding.

Customer acquisition. Data mining is used to help improve customer acquisition efforts by identifying the profile of potential buyers for a particular product or responders to a campaign. While these derived profiles can lead to improvements in marketing efforts, one can only infer the reasons these groups respond where others do not. With market research one can survey customers to understand why they buy a particular product or respond to a specific campaign. Used together, data mining and market research can provide more actionable results in a more efficient manner. Specifically, data mining can identify customer segments to survey and provide hypotheses as to purchase intent and market research can narrow field work to a tighter segment and more focused research objective.
Customer retention. Market research is well equipped to identify drivers of satisfaction and loyalty. By matching primary market research data to a customer data warehouse, data mining can be used to identify behavioral links between reported satisfaction and loyalty. Additionally, data mining can be used to validate a relationship between reported loyalty and actual churn behavior. Used together, data mining and market research can more accurately identify key drivers of customer loyalty and enable an active management of customer churn.
Cross-selling. Data mining is often used to identify naturally occurring associations between products. Marketing managers use these associations to develop joint-marketing and cross-selling campaigns. However, many times product associations are not obvious or only occur within specific customer segments. Data mining is often ill-equipped to provide further insight into these patterns. In such circumstances, market research can be utilized to focus on what factors lead to these associations. This research can result in more effective cross-selling campaigns and product promotions.

Where should data mining and market research converge?

The convergence of data mining and market research can best be illustrated by examining the underlying research stages common to both disciplines. To this end, we define the underlying research processes as consisting of six distinct stages (Figure 1). These stages include:

define where the customer is articulated;
capture where information is collected;
store where information is managed and maintained;
analyze where information is examined;
understand where insights and conclusions are drawn;
deploy where insights are operationalized throughout the organization.

Data mining process

Data mining most commonly defines the customer as a set of trackable behaviors. This is due in large part to the fact that data mining requires large data sets. These are more often produced by operational systems than surveys. This means that the customer is defined as an acting entity with less input from intentions, attitudes or outside behaviors. Therefore, data mining focuses on capturing what is accessible via operational systems that interact with the customer.

These systems produce massive amounts of transactional data including purchases, customer service inquiries, Web visits, phone logs and more. The data is stored in large data warehouses. The analysis of this data requires highly scalable algorithms that churn through the data looking for common aggregate patterns. Customer understanding is derived from interpreting behavioral patterns. Intentions are then inferred from actions. Finally, the insights gained through data mining are represented in the form of models that can be used to score databases and real-time applications.

Market research process

Market research defines the customer as a thinking, affective entity where intentions and attitudes are more important than actions. Market research often defines the customer as a group within the general population. Being freed from the internal corporate database, market research is able to explore questions such as competitive product assessments, intentions to defect and general satisfaction. The data outputs are subjective comments and ratings. The data is often captured in the form of spreadsheets or text files and delivered in the form of written reports. The analysis of this data is a subjective summary of the results and interpretation of meaning across the responses. Customer understanding is gained by linking the attitudes of general population segments to the assumed makeup of a client’s existing customer base. Deployment of market research results occurs through presentations to decision makers.

Combining processes

Combining data mining and market research will require synergy at each stage of the research process. While the customer deserves to be seen as a thinking and acting entity, combining these disciplines provides the unique ability to analyze the gaps that are known to exist between espoused plans and practice. Thus data capture must expand to include all information, subjective and objective, intentions and actions. The storage of data must come together so that the analysis stage can leverage both. In addition, the analysis stage must leverage new processes that take advantage of the best of both disciplines, including empirical behavioral modeling and qualitative research methods. Finally, the deployment of insight, whether to human or machine, should take advantage of the knowledge gained from both data mining and market research. Only when a full perspective of the customer is available can holistic conclusions be drawn and the most accurate insight can be deployed. (For a more detailed examination of the convergence of data mining and market research practice, see Convergent Research Patterns [Kenning Research Inc., 2003].)

Why haven’t data mining and market research converged?

Despite their shared fit within customer intelligence, their commonality of application, and their similarity of research stages, data mining and market research have still not converged into a unified research environment today. Systematic convergence has been hindered by several factors. Among the most challenging barriers to convergence are separations between data mining and market research with respect to organizational structure, culture and infrastructure.

Organizational separation

In most organizations today, data mining and market research operations are housed within different parts of the business. This physical separation hinders interaction and cooperation. Organizational separation also implies that two decision-makers, both tasked with customer intelligence, are operating under different strategies and objectives.

Cultural separation

The cultural separation between data mining and market research can be seen from the executive and field level. At the executive level, there tends to be a decision-making culture that is based more heavily on either internal analytics or market research. The comfort of decision makers toward one approach over the other perpetuates the separation of disciplines.

At the field level, there may exist an adversarial relationship between data miners and market researchers. This atmosphere of non-cooperation hinders the advancement of research.

“Anything where a person’s identity is used isn’t market research, it’s spying…We [market researchers] are always at risk of getting a bad name from people who mistake market research and data mining, which is about finding out enough about people to sell them something.” — president of a market research society

“What we need is not market research, it’s more transactional data. It is well known that past behavior is the best predictor of future behavior. Attitudinal research is weak at best.” — data mining expert

Infrastructure

Today, market research and data mining rely on separate internal infrastructures. Bringing these two disciplines together will require the integration of technologies that are not widely integrated today. Such technologies include data collection, data management, data storage, data analysis/reporting, and deployment. As well as general applications such as project management and knowledge management.

What are the benefits of converging data mining and market research?

Maintaining two separate disciplines for consumer research, data mining and market research leads to:

non-optimized use of available data;
non-optimized use of new learning;
redundant treatment of similar research questions;
sub-optimal conclusions drawn when one discipline is used where the other would have been more effective;
ultimately, the potential for non-optimized intelligence at a higher cost.

Organizations that commission data mining and market research are often rich with data. In many cases, data mining and market research can be improved with the inclusion of data generated for use by the other discipline. Bringing these two research areas together can lead to the identification of available data, which can be leveraged to derive deeper, more accurate insight.

By not converging these disciplines there is the risk that knowledge gained from one research initiative isn’t shared with the other. This can lead to the formation of conclusions that could have been improved by previous learning.

Certainly, an organization would want to avoid a situation where both disciplines are being used in an uncoordinated manner to address the same research question. For example, it is not uncommon for organizations to commission market research agencies to study the issue of customer loyalty, while in another initiative they have commissioned data analysts to develop models of customer retention. This is a good example of each discipline providing a unique and valuable contribution to the research question. Yet, the results will be sub-optimized and more expensive if they are not coordinated.

Recommendations

The convergence of data mining and market research may not be the best strategic initiative for your company at this time. Only those companies who today are making a significant investment in customer intelligence and market research can expect significant gains from convergent research. If your company collects behavioral and attitudinal data on your customers, has numerous customers with whom you engage frequently, and is under competitive pressure to grow and maintain your customer base, consider the following recommendations.

Determine the need

The first step toward the development of a convergent research environment at your organization is an internal assessment. Review the following questions with relevant individuals within your organization. Do you commission market research and data mining today? Are they being conducted separately? Are they being conducted to address similar business questions? Is customer intelligence critical for business operations? Can incremental improvements in customer intelligence result in significant advancements in business performance?

Test the readiness

Examine your internal data mining and market research operations. Distribute this article and get their reaction. Assess the cultural readiness of your team to adopt a convergent research discipline. Examine the organization structure that houses data mining and market research. Develop a chart that documents the relationships among those who are pivotal to the research process for each discipline. Identify the cultural and organizational barriers that separate these disciplines. Be sure to document the strengths and supporting relationships as well. Review the technologies each group utilizes to perform its research. Identify the overlap and differences in the required infrastructure. Determine if these information environments can be coordinated.

Then, identify all internal consumers of market research and data mining results. Interview these decision makers to understand how they use these streams of information. Ask them how they synthesize this information in their own minds, and what they desire from the research, and assess the potential business benefits from convergent knowledge. Also, identify all non-human deployment of research results, whether these are in the form of scores back into the database or recommendations to real-time operational systems. Assess the potential benefit of improving the accuracy of these scores, even by the smallest amount.

Start small

While it is important to have a vision of what is possible and how to get there, take one successful step at a time. Pick a small pilot study to measure your internal readiness for the convergent research paradigm. This pilot should test the organizational, cultural and technological environments. It should also be designed to demonstrate the “lift” generated as a result of convergent knowledge compared with traditional research approaches alone.

Start strategic

Due to the initial investment in the pilot, choose an application that has high strategic value. Or pick a tactical application that has the promise of high financial returns. Chances are, a successful pilot project will lead to the identification of a larger implementation of the approach. Proving the concept on a highly visible and strategic application will insure greater excitement and buy-in for further progress towards convergent research.

Find support

All change, no matter how beneficial, is difficult. It will be easy to slide back into the old ways of doing things. Choose a consultant, under a limited and focused engagement, to help you through your internal assessment and to help design a pilot program. Make sure the consultant has experience in both data mining and market research toward improving customer intelligence.

Share your success

Find a forum to share the results of your successful initiatives. Not only will this establish you and your organization as innovative and adaptive, it will foster the development of a supportive community of like-minded contemporaries who will challenge each other to grow and refine the discipline of convergent research. Start now and you can develop and maintain a convergent research practice as your competitive advantage.