Expert opinions
Editor’s note: Jeffrey S. Robbins is CEO of Database Sciences, a Paramus, N.J., research firm.
Recently, our firm was hired by Soliloquy, Inc., an Internet infrastructure company centered on natural language understanding, to help fine-tune an innovative product that serves as a front-end interface for online vendors. The product is a natural language Expert which allows shoppers to find what they want through a two-way interactive online “conversation.” Additional value is brought to the vendor in that the software maintains logs of the dialogue and Soliloquy can then mine them for insights into shopper psychology.
The Expert replaces the typical database/shopping cart e-tailing application where shoppers type in search criteria, look through the results and make buying decisions. Instead, shoppers engage in a conversation with the Expert and the software processes the typed dialogue to extract relevant words and phrases so it can search a product database accurately.
We were asked to help evaluate an Expert on laptop PCs. Essentially, when users access a Soliloquy-enabled PC store via their Web browser, they are greeted by the Expert. The Expert asks what the shopper is looking for in a PC and makes it clear that any question can be typed in. As a guide to the shopper, there are sample questions available to look at, parameters by which one can compare models (screen size, RAM installed, hard-disk size, etc.), as well as prompts from the system to help narrow down choices (i.e. How much do you want to spend?). Additionally, the system is capable of explaining technical terms to the user (i.e. What is RAM?). As a user defines his/her needs by expressing choices in differentiating criteria during the conversation, the Expert updates the shopper on how many items in the database are possible matches. At any time, shoppers can change criteria, look at details about matching products, or further chat with the Expert to narrow possibilities and/or learn more.
Our job was to put the Expert through its paces and provide insight into whether it could be a preferable alternative to existing online shopping cart interfaces and/or live salespeople in stores. Clearly, a key to the Expert winning out over salespeople was in its ability to accurately process the natural language dialogue. As such, the client wanted to compare opinion data (collected via survey) with Web observational data (collected in Soliloquy’s natural language logs). By doing so, we could possibly trace dialogue paths and bring much more meaning to comments such as “I love the Expert” or “I was frustrated.”
Fielding the study
We recruited respondents with varying amounts and types of online shopping experience to participate in the study. Additionally, we were mindful of several demographic variables so that we could evaluate their possible influence on a respondent’s experience with the system. Once qualified through an online screener, respondents were sent to an online demo of the system and given a shopping task to perform. Afterwards, respondents were sent to a post-demo survey so that we could get their feedback on the experience.
From our perspective, the most interesting aspect of the study was the combining of Web observational and opinion research. From a technical standpoint, the study presented a challenge. We typically host surveys on our own Internet servers driven by our proprietary Web survey engine, but the demo had to be run on our client’s server (so that they could capture and evaluate all of the natural language interaction between the system and the respondent). Thus, our system had to pass off a unique identifier to the client system - which then had to pass back the identifier to us for the post-demo survey.
From a research standpoint, we see great value in the methodology of sending respondents through an online exercise, immediately capturing their opinions, in addition to their Web behaviors, and then analyzing the different data types together.
Often, we see Internet companies only looking at what is occurring on their sites, and not exploring why. Certainly, its important to know how many hits a site gets, and what the ratio of purchasers to hits is. But what’s behind the numbers? Would the purchase ratio be higher if the Web shopping interface were more intuitive? Or, conversely, is the interface less of an issue and the purchase ratio more dependent on price and/or competitors’ actions?
On the other hand, of course, market researchers spend their days helping clients learn about the opinions and motivations of their customers and potential customers. In many instances, opinions about an event (whether it is a shopping experience or making a decision on a brand purchase) are being solicited well after the fact. Time can be our enemy in learning precisely what factors influenced a consumer’s choice. In the brick-and-mortar world, to be able to do what we did for Soliloquy would require sending camera crews and interviewers around with respondents to capture the shopping process. Online, it’s a much simpler proposition, as virtually every move we make on the Web is already tracked, for better or worse. Aside from the technical requirements, the only other hurdle we had to overcome was in designing the exercise to fit within the environmental constraints of conducting research over the Internet (time, attention span, etc.).
Clearly validated
In the end, we were able to help Soliloquy learn a great deal about consumer perception of their product. First, and foremost, the concept of the Expert natural language shopping interface was clearly validated. We did, however, discover some usability issues that needed to be addressed, as well as natural language understanding trade-off assumptions that required some further evaluation. Because of our methodology, the client was able to examine each respondent’s behavior, compare it with his or her stated perceptions, and categorize the overall experience. Leveraging this information, the Expert product can be tweaked further, based on the target market and product line of a particular vendor.
Conducive to learning
Experienced researchers understand that Internet/Web based research cannot always replace traditional market research techniques. In fact, for most businesses, online research should carefully be evaluated for its applicability in the overall research methodology mix. For online business managers, however, the environment of the online marketplace is conducive to learning a great deal about customers and visitors. By comparing survey research results with site data they already collect, e-managers can effectively explore consumer behavior, perception, and their dynamic relationship.