Does UGC make you say 'Ugh'?
Editor’s note: Andrew Wilson is senior project director at Applied Marketing Science Inc., Waltham, Mass.
More people than ever actively participate in blogs, forums, product/service reviews, social media sites and other platforms for user-generated content (UGC). The opinions expressed in these venues are unsolicited, honest, candid and passionate and can be extremely thoughtful. The topics are varied and people discuss everything from what they did last night to the technical nuances and likes and dislikes of the latest gadget.
With the explosion of UGC, managers have begun to face the question, “Are you listening to UGC?”, but how you listen will dictate what strategic decisions you can make with this new data source. The UGC ocean is vast, but having a lot of data does not automatically imply you can or should analyze these data quantitatively. UGC should be viewed as a series of unstructured conversations that, when used appropriately, can add depth to and expand your understanding of who your customers are and their experiences with your products and services.
Many quickly find the sheer volume of online conversations overwhelming and look for tools to help focus their efforts. One popular solution, dating back to the earliest days of UGC in the late ’90s, is to “scrape” the Web and use text analytic tools to automate the analysis. How does scraping work? The researcher selects a number of search terms, much like a search engine, and then the tool crawls the Web in search of UGC relating to these search terms. All positive hits are then aggregated in a database. Next, a text analytic tool counts how frequently a product, feature, brand, etc., is mentioned and makes an attitudinal assessment (e.g., positive, neutral, negative) for each statement.
This solution is relatively inexpensive and enables reports to be generated quickly. But there are some issues. While these tools make the task of “listening” easier by automating searches and distilling the endless search results into quantitative data, there are three inherent problems that people should be aware of when collecting and analyzing UGC quantitatively: representation, contextual confusion and online echo.
To demonstrate how these issues can be disruptive, consider the following fictional example. Imagine a manager at Toyota wants to use UGC to investigate whether consumers feel the product quality of Toyota’s Scion is superior to a model offered by Mazda. Let’s assume that after scraping the Internet, an automated program reports that 70 percent of all the UGC relating to both the Scion brand and its product quality is positive. Similarly, this same tool finds that only 50 percent of the comments relating to both the Mazda brand and its product quality are positive. The manager might initially conclude that consumers believe Scion has higher product quality than Mazda, but let’s dig a little deeper into these numbers by asking a few simple questions about the data.
Representation
First and foremost, who is responsible for that 70 percent and 50 percent? Are those figures representative of all Scion and Mazda customers? If not, are they representative of the same subset of customers (e.g., auto enthusiasts, actual customers, potential customers, etc.)?
The Internet users responsible for user-generated content are not necessarily representative of the general public, a company’s customer base or that of its competitors. Much of today’s UGC is created by a relatively small minority. There is also a possibility that different types of customers are responsible for UGC for different companies (e.g., young auto enthusiasts may be primarily responsible for Scion’s UGC while young professionals could be predominantly responsible for Mazda’s). In 2009, Forrester Research found that while Internet penetration is approaching 80 percent, only 24 percent of adults would be considered creators - those who publish a blog, publish a Web page, upload videos, upload audio/music they created, write articles/stories and post them (Source: Forrester Research’s 2009 Consumer Technographics Study).
For most UGC, the common thread is passion. People are driven to blog, post, review, tweet, etc., because they are passionate about an activity or they experienced an abnormal or extreme experience that they’d like to share with the world. Without understanding who is responsible for the UGC behind the above figures, one should be cautious before concluding that consumers or even certain consumer segments perceive Scion’s product quality to be superior to Mazda’s.
Contextual confusion
What percentage of the results is truly relevant to Scion, Mazda and their product quality? What percentage of the results was misclassified as positive, neutral and negative? Are these percentages the same for Scion and Mazda?
Context is essential to understand meaning. It is also extremely difficult to automate since contextual clues are constantly evolving alongside language and are often very subtle (e.g., discerning Apple the company from apple the food). While text analytic and Web-scraping tools are improving in their ability to understand context, it is certain that any Web-scraped database still includes many irrelevant search results and excludes some relevant ones. You can also be sure that it has assigned improper attitudinal assessments to many of the phrases. Currently, human oversight and review of the data are the only ways to be sure your data are truly relevant to your search terms and that attitudinal assessments or other codes are correctly assigned.
Here are some examples that highlight the contextual confusion issue. Most of today’s text analytics tools would classify the following statement as a negative comment about the Scion: “With the supercharger included on my Scion, it is one bad machine,” not being able to recognize the slang use of the word bad to actually mean good. Similarly, most of the technology would classify the following statement as being positive for Scion: “I loved my Scion then I had to replace the transmission twice in the past two years.”
Further complicating matters, much of a Web-scraped database will consist of forum posts and replies, which have complicated references that are easy for you and I to understand but extremely difficult for a computer program. For example, when visiting a Mazda enthusiasts’ site you might encounter a post titled, “My suspension is amazing,” where a user gushes about how great the suspension is on their Mazda 6, which is then followed by replies such as, “I agree, but my previous Mazdas had poor suspensions,” “I’m not sure I agree with your earlier post, that car really has oversteering problems,” “You’re wrong, that system is horrible,” “With the new upgrades it’s much better than stock and now I love my car,” etc. Imagine Mazda wants to use these scraped data to evaluate how consumers feel about the Mazda 6 and its suspension. You quickly get a sense of how language and context can confound an automated program that is determining relevance and providing an attitudinal assessment.
Online echo
Of the results, how many are duplicate thoughts from a single user? Is this equivalent for each brand?
Internet users creating UGC rarely limit themselves to a single outlet (e.g., forum, blog, Facebook, etc.). Thus, search results often contain a single person’s thoughts expressed on multiple sites. Unless the wording used on each site is identical, the database cannot identify the person as the same user and remove these duplicates. Most UGC is anonymous. While there is technology that can scrape data and attribute results to a single user, the universe of user-identified UGC is dramatically smaller than the user-anonymous universe. Consequently a scraped data set is likely to be distorted by very vocal users expressing themselves multiple times in several different venues. Understanding the degree to which duplication dilutes your data can be a guessing game.
Consider the following hypothetical example. A customer has their transmission fail but the failure was not covered under warranty. They post about it on their blog and then again on an auto-enthusiast forum to vent some more. Later they discover that others have similar experiences and decide to tell their story to anyone who will listen, hoping to warn other consumers and/or warrant a response from the manufacturer. These events are not uncommon and, more importantly, disrupt quantitative analysis of UGC data.
These three issues - representation, contextual confusion and online echo - make quantitative analysis of UGC precarious. While some academics have used UGC quantitatively to successfully predict movie box-office success and stock prices in the short term, those applications have been unique and difficult to apply to other applications. This is not to say that quantitative analysis of UGC cannot be useful, but people should be aware of the data issues that exist and adjust their strategic decisions accordingly.
Sift through the mass
A more robust method for analyzing UGC is to use Web-scraping and text analytics tools to sift through the mass of UGC, but to then analyze the results qualitatively. UGC can be incredibly influential, but understanding what people are saying - instead of how many people are saying it - is where the real value lies.
Rather than beginning with UGC and letting it drive how you listen to customers, a better approach begins with identifying the important strategic issues your organization faces. Then determine whether collecting UGC will enhance your understanding of those issues. Are people even talking about your product or service category online? A quick search on Google, Omgili or Technorati can give you a sense of what’s being discussed.
In deciding whether or not to analyze UGC, an organization should also consider if an important segment’s thoughts are absent from the UGC universe. If they are, can others speak for this segment (e.g., products targeted toward the elderly)? If not, it may be necessary to supplement the Web-scraped UGC database with an additional form of exploratory research (one-on-one interviews, focus groups, etc.) or consider a different approach entirely.
Let’s return to our earlier car example to demonstrate how an organization could analyze UGC more effectively. Imagine that an automotive manufacturer wants to investigate what wants and needs customers have when purchasing a vehicle. Feeling confident it has not excluded important opinions or groups, the organization might scrape the Web, looking for UGC to understand how customers evaluate automobiles (e.g., how customers define product quality, what features they consider, current issues with vehicles today, issues in the buying process, etc.). This broad search would likely net far too much content for any one person or small group to read the entire database. So cleaning the database of irrelevant results should be the first step, which might involve removing any comments that are fewer than 10 words or comments about pricing and promotions, which tend to be abundant and irrelevant. There are other obvious topics that can be removed from the database but beyond that the task becomes subjective.
To give you a sense of just how much irrelevant data there could be in a scraped database, consider this actual example. During a recent experiment at our firm, we scraped the Web for UGC on mouth-care products and found that only 10 percent of the results were actually relevant to our topic.
Hit the mark
A word of caution, which may seem obvious: the more refined the search, the smaller the resulting database. While this makes sifting through content easier, it simultaneously increases the risk of removing key insights. Striking the right balance can be difficult, but you’ve hit the mark when the majority of the cleaned database is relevant and an analyst can actually read through the entire database. While this may take a while, they’ll still be able to pull out lots of nuggets of useful information.
Next, using text analytics, an analyst can classify comments by features, issues, buying processes, etc. It is also useful to understand which features are frequently talked about together (i.e., co-occurrence) and many vendors offer tools that automate and graphically display these results for your UGC database.
Using text analytic tools this way and graphing the co-occurrence of topics helps organize the content so that you can learn faster and more effectively. Classifying the cleaned data makes it easier to see broad trends and understand issues more completely. As you continuously read through the database you’ll discover new classifications and important issues you initially overlooked. The goal here is to develop a qualitative report similar to one you would receive after conducting one-on-one interviews, focus groups, ethnographies or other forms of qualitative research. This process will help you identify key issues, develop theories about customers’ experiences and understand their wants and needs.
Given a structured database of UGC, it is tempting to try to reach conclusions about the features and issues that are most important based on volume and an attitudinal assessment, but this process should be reserved until later. Often what is top-of-mind may not be most important and, given the issues discussed earlier, there is potential for your analysis to be misleading.
To make judgments about what features and issues organizations should focus on or which ones are most important, a more rigorous quantitative research method would be the logical next step. This could involve asking a random sample of customers to go through an importance and performance rating exercise or possibly a conjoint exercise. These more rigorous quantitative methods will provide a much clearer picture about what to focus on and the organization can have much greater confidence in these results. With appropriate experimental design and sampling methodology, this approach ensures the results are representative and are not susceptible to the issues discussed earlier.
Extraordinarily influential
As online outlets (blogs, social networking, reviews, forums etc.) have exploded, customers have become more powerful in their ability to influence each other and the businesses they interact with. Even though UGC may not represent the broad views of the market, it can be extraordinarily influential, given its ability to reach others. Therefore understanding what is being talked about online and the implications of those discussions is crucial for many businesses. By understanding the limitations inherent in some of the UGC data collection and analysis techniques currently available, organizations can identify appropriate times to use these data and when to seek out more robust insights.