Editor's note: Pamela Walshe is director, user experience research, at AnswerLab, a San Francisco research firm. She can be reached at pwalshe@answerlab.com. This article appeared in the April 22, 2013, edition of Quirk's e-newsletter.
Research methodologies must evolve constantly to keep up with today's changing technology and user experience (UX) research is no exception. Regardless of the industry, platform or phase of product development, UX researchers must broaden their toolkits to include creative solutions that go beyond the standard go-to usability tests for design teams. The most successful product teams use different methodologies throughout the product life cycle, as each approach brings a different lens to the questions they are trying to answer and therefore generates unique insights.
User experience research is critical to ensuring overall success of an experience so I'd like to share a quick cheat sheet covering some of the most effective methods that you may not have tried, including what they are, when to use them and why you should.
Expert reviews
What: Seasoned UX researchers evaluate your designs or concepts against a set of heuristics (design best practices) and provide feedback and recommendations for improvements.
When: Great for low-fidelity concepts or prototypes to get insights early enough to impact key design decisions; also a great option if timelines don't allow for user testing. In the example below, a design team was struggling with 10 potential design solutions on a mobile app. The expert review helped the team identify three clear front-runners that were further iterated and ultimately tested with users. This method is popular with design teams looking to optimize their designs prior to launch or to find out what is or isn't working well with a current design.
Why: Research experts can identify many of the usability issues that would be uncovered in usability testing and can get you the insights more quickly than conducting primary research because there is no recruiting phase to the project - saving time (roughly two weeks) and money.
Wait, there's more! Experts share examples of how others may have solved UX challenges, providing insight into best practices within and outside of your industry.
Online focus groups
What: A researcher leads a group discussion with participants online. The online tool enables participants to see and interact with each other and react to visual stimuli.
When: Focus groups are a great tool to understand attitudes and beliefs around concepts that could influence adoption. For example, if you wanted to explore the barriers and motivators to using mobile payments, a group discussion would yield a laundry list of use cases, opportunities and barriers. This method is popular with product managers or marketing professionals seeking strategic input into the product roadmap.
Why: Online focus groups have the added benefit of reaching a broad audience across a wider geographic range without requiring travel. With the ubiquity of Webcams, it is relatively easy to recruit most user groups for online participation and the benefit of participating in their own home means that participants are often more comfortable and therefore more forthcoming than they may be in a lab setting.
Wait, there's more! Tools like online polling enable the researcher to pose a question to the group and elicit feedback privately before sharing with the group - capturing individual responses before having a wider discussion. A whiteboard feature also encourages interactivity. For observers, online chat (not visible to participants) helps create a "back room" experience so that you can share your observations and thoughts as a team while the research is unfolding.
Participatory design
What: Participants are led through a series of exercises whereby they use markers, sticky notes, labels and images to illustrate how they conceptualize information or processes.
When: If you really want to attempt to get inside the minds of your users, there really is no better way. In the example below, an automotive team was trying to understand how people think about the process of configuring a car and where they expect to find key calls to action. Interaction designers like this method because it helps them think through the architectural issues and overall information design of task flows.
Why: Especially helpful when designing process flows, participatory design can get tacit knowledge or beliefs out of the head of the user and onto paper. This is most critical if designing a new digital experience. It can help design teams understand the logical order of steps within a process and to know when and where to position calls to action. In short, it can help to avoid costly design mistakes before getting too far down the development path.
Wait, there's more! The highly engaging format of these studies is exciting for both the participant and the observers; participants can think more freely with a blank slate in front of them instead of reacting to a more designed experience. In this way, more creative ideas often surface. Observers benefit from seeing the visual artifacts created and identifying patterns created by real users.
Diary studies
What: Participants self-report their activities and experiences with a product or service over a given period of time. Diary studies can incorporate a wide variety of participant feedback methods, including survey responses, photos, videos or short phone messages. The output is a set of data and artifacts that give visibility into people's routines and habits.
When: Any time you need to learn more about the habits and routines and natural use cases for your products. They are also a useful tool to capture insights at specific phases during the user life cycle (e.g., a new or returning customer's experience). Many product teams use the data collected from diary studies to form the basis of personas.
Why: Diary studies provide you with context as well as an opportunity to observe and discover details that don't always come up in the lab. Diaries are particularly helpful in understanding mobile behavior, since the physical environment is such a big influencer on people's activities and therefore can't be observed in a traditional lab environment.
Wait, there's more! The artifacts that are collected through diaries - including photos and stories - are powerful internally within the organization to help illustrate common behaviors and to give visibility to users' environments. It helps to ground product teams in the real experiences of actual users as they work to evolve the product over time.
Mobile surveys
What: A live pop-up intercept survey on your mobile site lets visitors evaluate their experience and deliver insights into needs, expectations, behaviors and site performance. This approach offers flexible, real-time feedback and survey results that can be integrated with non-survey data for cause-effect analysis (i.e., impact of a new design, advertising, promotions, etc.) and provide direction for ongoing enhancements.
When: Mobile surveys can be done on a continuous or pulse basis, depending upon the marketplace and competitive context. We recommend the continuous approach when the marketplace is dynamic and competitive; there is a need to associate in-market effects with the survey data; and there is heavy site traffic.
Why: Consumers today are visiting mobile sites for far more than just a quick check-in or update. They expect mobile sites to be fully functional, to quickly browse and search with limited scrolling and they expect assurance of security. It is crucial to quantitatively measure the user experience on a mobile site to understand behaviors on the channel, track KPIs, ensure the user experience is optimized and identify pain points and strategies for ongoing enhancements.
Wait, there's more! The technology to measure the user experience on mobile sites is rapidly evolving, with more user-friendly, robust survey techniques. We can not only assess the site's performance but do so in real time while understanding the tasks performed and relate to the actual situation in which the site is being used (i.e., at home, work, while shopping, etc.).
Remote mobile usability testing
What: Usability testing conducted remotely with participants on their own devices in the comfort of their own homes/offices. Observers can view remotely and watch everything the participant is doing on their mobile device.
When: Usability testing is a critical phase of a user-centered design process, as it is often one of the final stages before launching a new product. Usability testing is also sometimes used prior to a redesign to better understand what is currently working or not working on a live design. The unique aspect of this approach is that it is conducted remotely, on the participant's own device. Note that this method requires a fairly tech-savvy research audience so that should be taken into consideration to ensure this aligns with your target users.
Why: While mobile usability testing can be readily conducted in a lab environment, the option to conduct testing remotely is important when you need to reach a low-incidence audience or when you explicitly need to reach users in distinct geographic markets. With remote mobile testing, you can reach users wherever they are.
Wait, there's more! By conducting research remotely rather than in a lab, costs are generally reduced. This gives you the option to save the money or to invest the extra you would have spent on recruiting additional participants to get a larger sample size.
Eye tracking
What: Eye tracking enables us to capture the gaze of participants as they interact with a Web site while also tracking clickstream data to illustrate the visual attention and behaviors of site users. Heat maps (seen below) and gaze plots provide a visual map of "hot" areas of a site and also clearly show page elements/areas that are given limited/no visual attention.
When: Visual design teams use eye tracking when trying to precisely lay out page elements and to ensure the visibility of business-critical page elements (i.e., calls to action, purchase buttons, lead forms, etc.).
Why: This method can be eye-opening for teams trying to drive very specific and discrete user actions. For example, a client recently used eye tracking to optimize a lead form on its home page. We learned that a photograph that was co-located next to the form fields dominated the visual field, reducing lead form completion. After removing the distracting image, the form fields became the focus and leads immediately improved.
Wait, there's more! The heat maps produced with eye tracking provide compelling visual evidence to a team about what is and isn't being seen by the target audience. Additionally, eye tracking provides insight into common patterns for viewing the on-screen information via gaze plots, as well as first-clicked links and click paths through the site. This is especially useful in identifying unforeseen dead ends or opportunities to optimize navigation overall.
Continues to expand
As I've outlined above, the UX research toolkit continues to expand as new technologies and best practices evolve. I hope this cheat sheet inspires you to step out of your comfort zone and try something different the next time you embark on a UX study. Over time, a multimethod approach ensures you're getting the most holistic view into the user experience you're creating and helps deliver a great experience for your customers.