Beyond the heat map
Editor’s note: Mike Bartels is senior project manager at EyeTracking Inc., San Diego.
There is no doubt that the eye-tracking revolution has begun. What was once an undersized niche market, comprised of a few early adopters and university labs, is now used in many diverse industries. Recent advances in hardware have improved the flexibility of eye-tracking systems and new software has eased the process of data collection. These enhancements, coupled with a growing interest in exploring more objective methods, have inspired a new generation of researchers to investigate this technology.
Of all of the industries that have begun to use eye-tracking, none have been more enthusiastic than the fields of usability and marketing research. Advertisers, Web developers, package designers and media directors have all started looking to the eye of the consumer for insights into product perception and motivation to purchase. The potential applications of eye-tracking - from the television screen to the computer monitor to the grocery-store shelf to the billboard - are myriad.
However, for many in these fields, initial attempts to put eye-tracking into practice have been less than satisfactory. Data have caused confusion. Findings have seemed inconclusive. Problems have been left unsolved. Lost in all of the excitement of running that first eye-tracking study has been a very basic question: What does it all mean? Standard eye-tracking analysis software, which generates little more than a heat map of visual attention and some simple viewing percentages, demonstrates a problem of style over substance. Sure, the heat map possesses a great deal of “Wow!” value, but what can it really tell us about specific research questions? How does this tool get us any closer to understanding our potential customers?
These are reasonable concerns. But before researchers transform their eye-trackers into high-tech paperweights, it is worth taking a step back and considering the methods. A meaningful interpretation of eye-tracking data requires a specialized course of analysis, one that involves thorough understanding of visual behavior and how it relates to other information. As a starting point, we suggest three basic principles that one must consider when attempting to use eye-tracking for their next study.
Principle 1: There is no simple answer
Take a look at Figure 1, which shows the visual attention to a Web page. What does this tell you? Is the Web site successful? Do people understand the content? Can a new user find what they are looking for quickly? What features cause confusion? What features are most useful?
(Figure 1: Analyzing the heat map won't give you the full picture of a viewer's experience with and reaction to a Web page.)
If your answer to the preceding questions is “I don’t know,” then you understand the first principle of interpreting eye-tracking data: There is no simple answer. Anyone who attempts to evaluate a medium by simply generating a heat map of eye-tracking data has failed to appreciate the complexity of consumer behavior. Did a particular advertisement receive so much attention because it was visually appealing or was it because it caused confusion? Was the package on the shelf completely overlooked, or was it noticed briefly and then actively ignored? These kinds of complex questions are beyond the scope of simple graphic representations of eye data. A heat map is one of many useful ways to illustrate trends, but it should only be used to complement more descriptive assessments. The movements of the eyes are part of an intricate system that cannot be fully explained through simple analyses. Further layers of exploration are required to understand the root of visual behavior.
The first principle offered here dismisses the idea that eye-tracking is the simple answer to all of your questions. Sorry, you’ll find no marketing research panacea here. This does not mean that the conclusions will be complicated; in fact, a properly conducted eye-tracking study will generally produce results that are quite intuitive and easy to understand. This principle simply means that, in order to draw insightful conclusions, our methods of analysis must delve deeper into the user experience. As we’ll discuss in Principles 2 and 3, quantitative and qualitative resources must be used together to establish a closer connection to the consciousness of the consumer.
Principle 2: Quantify the consumer
A quantitative interpretation of eye-tracking data requires a working knowledge of statistical analysis and a full understanding of the many ways that eye data can be scrutinized. It starts with the initial setup of the study. If you are interested in learning how much visual attention is allocated to a logo in a commercial or a product shot on a package, you must recruit a large enough sample to ensure that you can draw statistically-significant conclusions. The specific sample size will depend on multiple factors, including how many different groups you are testing and how many designs you are showing. It is usually best to consult a statistician with experience in eye-tracking research to develop the optimal study design.
Other factors that should be considered to ensure the viability of quantitative results include the order of presentation and the experience of respondents. It does little good to report that a golf-course advertisement received 11 percent of visual attention on a magazine page if the advertisement was always shown on the last page and only to people who dislike golf. Taking these steps early in the process of study development is well worth the effort when you can later boast scientifically-valid results.
After designing a sound research study and running participants, the real fun starts. There are many ways to analyze eye-tracking data from a quantitative standpoint. Most basic analysis software provides the option of examining the percentage of visual attention allocated to specific areas of interest. When used correctly, this is a powerful tool in demonstrating which features are seen and which ones are not. If you only want to know how much time people spend looking at your package on a shelf, then a percentage can tell you that. However, in our experience, marketing research questions are rarely this simple. The percentage of attention to a product, logo or advertisement does not generally reveal all of the necessary information. In order to see the big picture, it is often necessary to think small.
Figure 2 is a GazeTrace of one participant viewing a package for a period of two seconds. You cannot underestimate the value of thin slices in building accurate interpretations of eye data. By dissecting the testing sessions into second-by-second behaviors, a variety of new questions can be answered: What is the first thing that draws attention? How carefully is text considered? Which features are seen last? Which features are revisited?
(Figure 2: Analysis must be broken down into thin slices to obtain an accurate reading of what the eye movements indicate.)
Any broad analysis of a visual stimulus runs the risk of missing these precious morsels of information. As shown in this graphic, eye-tracking allows us to isolate and describe these small slices of behavior in a meaningful way. Web users do not experience a site or advertisement in one giant gulp; we consume it in small experiential pieces. Thus, analysis of our behavior should take a similarly-focused approach. A quantitative analysis of eye-tracking should aggregate these microfindings to clarify the more global conclusions demonstrated by overall percentages of attention.
To put it another way, you cannot describe the forest without looking at the trees. Luckily, for the trained eye-tracking researcher there are a variety of analysis options available to connect the minor details with the major themes. In addition to broad analyses of overall attention, there are focused methods for examining discrete trends. You can explore the order that particular features were viewed (e.g., was the product flavor viewed before the brand name?). You can determine how likely people are to look at specific content (e.g., how many people glanced at the contact information?). You can gather information on which items are viewed multiple times (e.g., once people viewed the advertisement, were they likely to return to it later?). You can establish the degree to which people are engaged by examining pupil dilation (e.g., did people zone out during the commercial or did they pay attention?).
Eye-tracking provides exact measurements for behaviors that were once only conjecture. Although the process of analysis is more complex, it pays off many times over in the quality and depth of learnings provided. Through quantitative analysis of eye data, researchers can add an objective component to a traditionally subjective field of study.
Principle 3: Qualify the eye
The human eye is an amazing organ. There is a great deal that it can show us in the way that it moves, the frequency of its blinking, the places it lingers and the dilation of the pupil. One important thing that it cannot do, however, is speak. Eye-tracking is by no means a substitute for a good qualitative interview. There are certain questions that can only be answered by directly asking participants what they think, just as there are certain topics that can only be addressed through analysis of eye movements. In our experience, when eye-tracking and interviews are incorporated into a hybrid research design, the quality of both components is improved.
One approach we employ uses information from eye-tracking to generate better interviewer questions and more accurate interviewee observations. This technique aims to combine the realism of an uninterrupted testing session with the depth of information available in think-aloud and focus group research. In other words, you can still gather the same detailed impressions and perceptions of the consumer without disrupting their experience by continually asking, What are you thinking now? ... How about now? ... And now?
The procedure is simple. Participants are allowed to interact with the testing material - be it a Web site, television show, package or magazine - without interruption. Once the interaction is complete, a video of their eye movements during testing is shown. As the participant watches their own visual behavior from the testing session, they can recall first impressions, points of confusion, positive features and other details that may otherwise have been absorbed into more generalized recollections. This technique has proven effective in combating some of the moderator’s most enduring vexations: respondents forget quickly, they make up stories, they add misleading details, etc. Providing the respondent with a video showing exactly what they looked at enhances the richness of their feedback.
We recently used this method to evaluate the advertising presence of an online university. After participants interacted with a Web page, they were asked why they had not clicked on any of the embedded advertisements. While considering this question, they were provided with images of the targeted ads, but eye-tracking videos were not initially shown. Most participants remarked that they did not click on the ads either because they never saw them or because the copy was unappealing. If we had stopped there, we might have recommended to our client that the advertisement be made flashier and the copy be reworked.
However, after following up with an interview, this original assessment proved to be misleading. While watching the video of their eye movements, many who at first claimed to have never seen the advertisements realized that they had actually looked directly at them. They were forced to resolve these differences between their subjective recollections and the objective measurement of their eye movements by thinking more critically about their experience. It became evident to most participants that they had, in fact, viewed the ads, specifically the animated graphic within each one. After noticing this, the moderator was able to focus a line of questioning on the impact of this graphic on their decision not to click. As it turned out, “unappealing copy” had been a useful scapegoat for respondents because it was easier to explain than the aesthetic problems that they had with the graphics. In this example, the use of traditional interview techniques alongside eye-tracking provided us with a means to move beyond respondents’ snap judgments toward a more accurate appraisal of their experience.
There are a few things to keep in mind when conducting this type of interview. The first is that the software used to replay the testing session must not be cumbersome. The moderator should be able to jump quickly between time segments in the video without disrupting the flow of the interview. When conducted properly, the recorded eye movements will make life easier for both moderator and respondent. Another point to remember is that you must remain cognizant of the larger picture. There is a wealth of interesting new qualitative information to be obtained from discussing participants’ eye movements but it is important to focus on major themes as well. The interview should strike a balance between overall impressions and the details from eye-tracking that drive perceptions. The final point is that this technique requires practice. To understand the nuances of eye movements, a moderator should prepare by watching videos of eye data and consider what different patterns of visual behavior might tell us about the thoughts and perceptions.
Quantify the behavior
In our experience, the three principles described here are essential for any researcher seeking to unlock the full potential of eye-tracking: you must be wary of oversimplification, you must quantify the behavior of your participants and you must incorporate qualitative feedback into your analysis.
This is by no means a complete list of everything you need to know to run a successful eye-tracking study, not by a long shot. There are other data to incorporate, such as Web usability measures and questionnaire responses. There are different types of hardware to choose from, including remote and headset models. There are multitudes of stimulus presentation methods, performance metrics, graphic rendering tools and analysis plans that all must be carefully scrutinized to determine the correct path for your study.
Obviously, getting started is no easy task, but the quality of results and clarity of conclusions make this effort worthwhile. A well-designed study featuring an elegant course of analysis has the potential to provide a valuable vantage point into the mind of the consumer. It is this potential that has stimulated excitement in the usability and marketing research communities.
The key to practically applying it lies in the training and tools of the researcher. Our experience has taught us that the three principles described in this article are a good place to start on your way to becoming a seasoned eye-tracking user. From there it is simply a matter of education, practice and perseverance. Whether you keep an expert on staff and conduct your research in-house or obtain eye-tracking from a professional firm, it is important to understand the basics of the technology.