How hybrid research allowed eBay users to give feedback on the feedback system

Positive, Neutral or Negative?

Editor’s note: Mike Katz is a principal user experience researcher with eBay Inc., San Jose, Calif. Peter Mackey is executive vice president, research for Invoke Solutions, Stamford, Conn.

Since eBay’s inception in the 1990s, its feedback system has been the primary means for the eBay community to police itself and ensure honest participation in the marketplace. The goal of the system was simple: encourage eBay users to evaluate transactions on the site for the benefit of the larger community.

EBay continually evaluates ways to improve the buying and selling experience and has made adjustments to the feedback system over the years to meet the changing needs of the marketplace. For example, as Figure 1 shows, eBay introduced “Detailed Seller Ratings,” through which buyers evaluate sellers on various specific performance dimensions such as whether their item was described accurately. EBay also tests feedback elements before they go live on the site to understand buyer and seller sentiment and larger effects on the community. Despite recent tests and other changes, the fundamental nature of the feedback system - based on a positive, neutral or negative ratings and including feedback comments - has remained relatively untouched.

In eBay’s continuing goal to encourage buyers to effectively evaluate seller performance and for sellers to understand how to improve their selling practices, in 2008 eBay decided to take a fresh look at feedback methods and explore changes that would potentially warrant fundamental deviations from current practices. The primary goals behind the changes would remain the same: to continue to allow buyers to effectively evaluate seller performance and for sellers to effectively understand how to improve their selling practices.

Represent the journey

Based on foundational research conducted earlier, a storyboard (Figure 2) was created to represent the journey of an eBay buyer and an eBay seller under specific use cases for a newly-proposed feedback system. The concept storyboard resembled a typical comic book but with interactivity made possible with Flash. With this storyboard, buyers and sellers would be able to get a primer on how the new system would address the central use cases of the feedback system:

• a buyer evaluating the trustworthiness of a seller;
•  a buyer leaving feedback for a seller following a transaction;
• a case where the buyer feels that the item was misrepresented by the seller;
• a case where the item arrived late or damaged;
• a seller evaluating feedback from buyers to understand and address areas for improvement.

Having created storyboards to represent the current feedback system along with the new proposed concept, eBay sought to understand buyer and seller reaction to the process, answering questions such as: Does it address perceived weaknesses of the existing feedback system? Does it introduce new problems? Would it have a negative impact on seller efficiency?

Concept testing with vision storyboards was important because it enabled rapid changes to the solution at an early concept stage, prior to design or engineering investment in high-fidelity user interface mock-ups or functional prototypes. While the temptation existed to skip the concept phase and proceed directly to creation of design mock-ups, project churn and the wasting of resources could have resulted.

Many concept testing options were considered but there were important weaknesses associated with each:

• Focus groups can be susceptible to group-think or bias associated with dominant personalities in the room.
• One-on-one interviews are impractical when input is sought from a representative sample of the user population.
• Surveys would require a tremendous time and attention commitment among respondents to provide feedback on all areas of interest and would not allow real-time consumption of user feedback or follow-up questioning by key stakeholders.

Optimal research solution

EBay determined that the optimal research solution was a hybrid qualitative-quantitative approach that allowed real-time consumption of data and deep-dive follow-up questioning with highly-engaged participants. To meet its research objectives, eBay collaborated with Invoke Solutions, a Stamford, Conn., research firm.

The Invoke platform (Figure 3) offered a hybrid research environment that blended an online survey with a discussion group. It also allowed real-time data analysis and on-the-fly follow-up questioning. EBay felt that the Invoke platform would provide the insights necessary to motivate future product improvements as well as garner support for the solution among stakeholders.

Expose the concept

EBay conducted research using the Invoke platform and exposed the concept to 140 eBay buyers and 180 eBay sellers in respective 90-minute live-moderated sessions. Participants logged into the Invoke platform remotely at a specified day and time to experience the concept storyboard and associated questioning. Participants could spend quality time (15 minutes) with the new concept storyboard on an individual basis. Unlike traditional focus groups, buyers and sellers were not privy to each other’s opinions (except where desired), avoiding the possibility of bias.

Specifically, participants were first shown the storyboard that represented the current system to refresh their memory of the details and to expose some issues revealed in prior research. They were then presented with the new concept storyboard and asked various questions comparing the proposed system to the current one. Following initial broad evaluation questions about the new concept, targeted questions were asked about specific aspects or use cases associated with the concept. Additional unplanned follow-up probe questions were also asked to dive deeper into participant responses (both quantitative [Figure 4] and qualitative [Figure 5]).

The goals of the investigation were twofold: gauge participant reaction to the concept overall as well as specific fundamental changes to the existing system; and understand areas for improvement for the next concept iteration.

Revealed specific issues

Evaluation of the new concept storyboard suggested broad support for the initiative but also revealed specific issues of concern for each user population. This research was then followed by an additional round of concept testing to ensure that the discovered issues had been addressed prior to further design and engineering investment. Based on the first round of testing, changes were made to the concept storyboard, including:

• elimination or simplification of sub-optimal solutions;
• introduction of new targeted solutions to address unsolved problems;
• a more in-depth rationale behind changes to the feedback system; and
• greater detail regarding proposed solutions in the original concept storyboard (to allow for more stringent stress-testing).

The second round of concept testing provided valuable information pertinent not only to the concept but to the broader effort involved in implementing such changes to the marketplace. For example, it reinforced the importance of adequately explaining the rationale behind changes to certain long-standing elements of the feedback system. (While the first round of testing suggested a lack of feasibility for certain changes, we found that providing a more detailed rationale behind such changes in the second round mitigated user concerns.)

We also realized that it was not possible to address every user concern. We discovered through more in-depth investigation that users disagreed with certain aspects of the proposed concept despite what we considered to be compelling rationales and solutions. This knowledge will allow eBay to focus appropriately on such issues when rolling out these changes to the community in terms of public relations, community outreach efforts and site messaging.

Together, the two rounds of concept testing provided the necessary insights to move forward with investigation of specific user interface design flows. As with the concept testing, iterative design research will be conducted to ensure the optimal user interaction and experience.

Garner support

In addition to providing valuable insight regarding buyer and seller reaction to a newly-proposed feedback system and how the concept might be improved, the real-time hybrid research approach also helped garner support for the initiative among key product managers. Product management executives attended a 90-minute session and came away more fully appreciating the value of the research and the central ways in which the concept could improve the eBay user experience. Plus, they realized areas for improvement that could drive future iteration of the process.

Additional benefits of the research approach included: deep customer involvement early in the product development process; cost- and time-savings over traditional methods; rich verbatims for better understanding of the data; no group-think or participant bias; rapid insight generation and follow-up; and an easily replicated process.

Yield tremendous benefits

Testing concepts early and iteratively in the product development life cycle (prior to implementation) can yield tremendous benefits that improve the likelihood of product success. The research methods described here provided eBay with a real-time qualitative-quantitative online platform that is playing a key role as eBay evaluates new methods of feedback.