Editor’s note: Jin Li is a software designer in the User Centered Design group at the IBM Toronto Laboratory. The author would like to thank his manager, Mike Fischer, for his encouragement, valuable feedback and support in this submission. He also wishes to thank the people who were involved in the evaluation sessions.
Competition in the marketplace is fierce, and having a competitive advantage often leads to the success of a product. In the past, having a leading-edge technology usually meant your product had the competitive advantage. However, as technology matures and reaches the point that it meets the needs of users, having a cool technology is often not enough. Today, usability differentiates good and successful products from bad ones. This is especially true for consumer products1. Leading software and consumer product companies are now promoting ease-of-use and integrated solutions to sell to a more mature market. Users are no longer simply impressed by features; they demand an integrated set of tools that solve their business problems.
To determine if our product has an advantage over our competitors, we perform a competitive evaluation. Competitive evaluation identifies areas where the prime competitive offering is most successful and areas susceptible to one’s own offering2. It determines the design strengths and weaknesses of the prime competitor. The output of this activity includes a prioritized list of the design strengths and weaknesses sorted by the degree to which they impact users’ tasks and satisfaction. This outcome is used to facilitate product decisions on how to effectively compete in the same domain. In addition, the marketing team can take the evaluation results and create documents for use in competitive marketing situations in the field.
Traditional competitive evaluation that uses a feature-by-feature, bottom-up comparison approach does not provide the vital usability and design information required in today’s competitive market environment. In addition, traditional competitive evaluations are costly in terms of time, budget and effort. It would be very useful if the existing competitive evaluation technique could be improved upon so that it would take less time to perform but would lead to meaningful information with respect to the product’s usability.
In this article, we propose a higher-level approach for performing competitive evaluation to gather qualitative and quantitative usability and design information by setting realistic goals for expert users to carry out the required activities. Contrary to the traditional evaluation method, our belief is that the whole is greater than the sum of its parts. Feature-by-feature comparison ignores the interdependencies among user tasks. Furthermore, users don’t perform tasks in isolation; they carry out an activity which is a set of interrelated tasks for a given goal.
Methodology
The essence of this methodology is to assess competitive products based on chosen metrics in a setting where natural task flows and real development scenarios would occur. Expert users are given a set of goals which define the scope of the tasks. It is important that this set of goals defines solutions to real user problems. Within our context, we consider "compile the Java source to byte-code" a task. We consider "1. Add a button; 2. Change the button label;" a description of a user scenario and task flow. Finally, we consider "Develop a Java applet with a push button labeled OK by hand coding" a goal.
Ideally, while the experts are using the product to solve the problem, they should not have to circumvent the product to optimize the solution. However, the frequency of such circumvention is a good indication of the deficiencies in the product. These workarounds and the differences in approaches taken while using the competitive products often provide great insights into potential redesigns and solutions to the usability problems discovered.
Expert users are given a set of goals along with the measurement metrics so that they can choose the most optimal path to solve the problem using the product. An analogy would be driving from point A to B. In the traditional evaluation method, drivers are given a map with a specific route highlighted from A to B and are asked to follow the marked route, even if that is not their natural or preferred path. In our method, drivers are given a map with A and B highlighted and are asked to go from A to B, with perhaps a few more constraints such as no more than 10 left turns. Each expert user is given a reasonable amount of time to practice their natural, optimal solution.
In the actual study session, expert users repeat their procedures for each goal three times. Full audio and video taping are done for both practice and actual sessions. The final metric for each expert is taken from the best (or average) of the three actual sessions. This methodology is different from the traditional competitive evaluation approach, since it does not impose artificial tasks or scenarios on the expert users. Experts are given the end goal and they choose the optimal path to accomplish the goal following their own natural task flow. Time to execute the competitive evaluation is a key advantage of this methodology. In addition, we can be more confident about these results than the results from traditional evaluations, since we believe that expert users employ realistic task flow.
They make fewer errors in this natural setting than they would when they are forced to carry out artificial tasks in traditional evaluations. Another benefit of this methodology is that one can obtain high-level information about expert users’ task flows and typical development scenarios. Expert users often embellish their tasks while they follow the optimal path to accomplishing their goals. Since only goals are defined, human-computer interaction (HCI) professionals who have domain knowledge must administer the sessions and analyze the results. Without such domain knowledge, facilitators will not understand why expert users perform the tasks as they did.
Applying the technique
We had the opportunity to apply this technique to compare Java programming tools that were available on the market. One of the objectives for our Java programming tool was that it be one of the fastest rapid application development (RAD) tools on the market. The experts’ goal was to develop a to-do list application using the Java development tool with which they were most experienced. The sub-goals, which the experts were asked to achieve in a stage-by-stage fashion, included the following:
- program the application visually;
- implement the application by hand coding;
- add a custom method to the application;
- debug the application;
- do the incremental edit-compile-debug cycle.
Since we wanted to know if our product was a faster RAD tool than those of our competitors, time to accomplish each sub-goal was used as the dependent metric. We found that higher-level goals are easier to identify than lower-level tasks and features, and are more readily agreed to by the product development team. These goals are akin to users’ typical problems encountered in their daily software development. Consensus among the team members and realistic goals are important for the validity and success of a competitive evaluation. While running the competitive evaluation sessions, real-world tasks and scenarios are brought out naturally by the expert users in the specific development context. Bottlenecks in the product design are easily seen, and given the right goals, high impact issues and problems can be discovered quickly.
Conclusion
We have described a new methodology for doing competitive evaluation to gather qualitative and quantitative usability and design information. Our recent experience indicates that it shows promise for comparing software development tools. We have found that we can obtain high-level qualitative information and achieve higher confidence in our results than when using traditional competitive evaluation.
We have used time as a measurement metric, but believe other objective metrics could be used as well. With our approach, expert users optimize their task flows to accomplish their tasks to achieve the given goals. The output from this methodology is more reliable and realistic.
References
1Norman, D., The Invisible Computer : Why Good Products Can Fail, the Personal Computer Is So Complex, and Information Appliances Are the Solution, MIT Press, 1998.
2Fischer, M., Expert User Heuristic Walkthrough: An Efficient Method for Assessing the Usability of Competitor Products, IBM internal publication, 1993.