Capturing smoke in the bottle

Editor’s note: H. Grace Fuller is the executive director and a senior faculty member of the RIVA Training Institute, Bethesda, Md. She is a member of the Qualitative Research Consultants Association (QRCA). The article below is an excerpt of her presentation at the annual QRCA conference in San Diego in October.

Human beings are the instruments of qualitative analysis, not computers, therefore the findings will differ from one analyst to the next. It is this ambiguity that is often a barrier to tackling the task of analysis.

In preparation for my presentation at the annual QRCA conference in San Diego in October, I turned to several of my fellow QRCA and RIVA faculty members and I asked:

  • What tips or techniques do you find useful when conducting analysis?
  • What’s a secret you’ve discovered?
  • What one thing would you tell a new moderator-consultant about qualitative analysis?

Summaries of their responses offer useful methods for pulling divergent qualitative information out of masses of data and capturing it in a form that gets used. I’ve included my own tips and secrets at the end.

Naomi Henderson, RIVA Market Research, Bethesda, Md.

“I have an idea in mind before I start analyzing, for example: What do I believe to be the ‘spine of understanding’ in this study? I realize that I can only report on the ‘tops of the mountains’ and not all the ‘interesting valleys.’ Therefore I analyze the items that directly relate to the study purpose first!

“A secret I’ve discovered is there is no time to do the long, detailed analysis that I would like. I have to analyze data for decisions-makers as quickly as possible. I understand that while researchers may see the analysis function as critical, clients see it as a ‘thing’ that we do. Another secret is that there is no magic software, no magic analytical tool, and no magic shortcuts to analysis of qualitative materials.

“I would tell new moderator-consultants to leave more time than they expect for analysis. I also recommend they write background data and the table of contents before they go on trips to lead the groups. Also, they should make sure the title is full and rich and explanatory, including the number of groups, types of consumers, and key content areas. Lastly, I suggest they set a structure for writing reports so their style is consistent from report to report.”

Alice Rodgers, Rodgers Marketing Research, Jemez Pueblo, N.M.

“What I’ve discovered about analysis is that it is done differently by different folks. Some listen to the tapes. Some have transcriptions. Some don’t do either. I virtually always have my tapes fully transcribed. My transcripts are all merged, so that I get a draft report back, with all the comments under each question, separated by demographics, for example, ‘some comments from the women in Portland,’ and so on.

“For me, having transcripts means that I don’t miss things which I think I would otherwise. However, for certain types of things (advertising checks, for example) it is not as critical. For new product research, I have found transcriptions to be virtually imperative. When I’m doing fast stuff that needs transcripts, I will have my secretary with me, transcribing on the spot - or on the phone listening and transcribing on the spot. When I’m doing super-fast stuff - a presentation-debrief almost immediately after the research - I have notetakers. I talk with them and also set up some forms for them to fill in so they know exactly what I need from them at the end of the evening.

“A tip or the secret of analysis is realizing that there are numerous ways to analyze and write reports. Find out what works best for you - and most importantly, provides maximum value for the client.

“The one thing I would tell new moderator-consultants is to get transcripts and, while working on the report, see if they discover things that they would have missed otherwise. I have discovered that one comment can be critical information and it could have easily been overlooked.”

Marilyn Rausch, Food Perspectives, Minneapolis.

“I suggest writing a one-page executive summary last, keeping in mind that this summary could very well be the only thing a key decision-maker reads. Make it clear, concise, and actionable. Write from two points of view: the respondent and your client. Make sure one of them is not your own.

“In your analysis, don’t miss the big picture for the details. Think about what you heard and didn’t hear, what you observed and didn’t see, plus the prevailing tone of the responses. Ask yourself whether the respondents were bored, pensive, impulsive, persistent, or wishy-washy. Think about aided versus unaided responses and the meaning of any difference between the two. When in doubt, put more credence in the unaided ones.

“Analysis requires a rested mind. Try to ‘sleep on it’ before you begin your mental processes in earnest.”

Dorrie Paynter, Leapfrog Marketing Research, San Francisco

“One thing I have found that really helps me with writing reports is if I do a ‘brain dump’ the morning after I’ve done some groups. I just put anything I can think of down on paper. It’s amazing how much you can really remember. Of course, you still need to go back and use the transcripts to write the report, but it helps me to remember the major themes I heard.

“Another thing I think is useful, and I have to admit I don’t always do this, is to write the report, write the implications or recommendations, then go to sleep on it and return to it the next day. Somehow separating myself from all the details for a night helps me get a perspective on what I learned and how it will help my client.

“The last thing that comes to mind for me on this subject is to review the objectives again before writing the report, and again before writing the conclusions. That helps focus the findings on the key issues.”

Vivian Thonger, Thonger Qualitative Research, Boskoop, Netherlands

“Several things spring to mind. First, take ‘hot notes’ after each group, however sketchy or unbalanced they may be. They should include your own view of how things are going, act as a dump for your own opinions, including your deepest private concerns about the clients’ motives, about their real purpose. Also add any group-specific vibes you feel or sense, oddities or unusual people or events happening around the group that affected the way it ran.

“I suggest taking notes after each group if you are dealing with many concepts and issues or statements. Make a grid of squares for the groups and concepts and in a square, sum up responses to key items. Just use checkmarks, crosses, or develop your own symbols. After all the groups are done you can easily scan the whole page to get the big picture.

“If you can’t rerun a particular group in your mind, listen to a segment of the tape for that group while you look at your hot notes. This should bring the group back to life and help you put your finger on the group dynamic.

“Reporting on your mood, vibes, or atmosphere where you feel it is related to the subject of the discussion gives the client exactly the added value they crave. It’s a source of emotional and sensory information. Content analysis is dry, dry, dry - and weak!

“Speed-read or scan all group transcripts. You can also write hot notes after listening to the tape of each group if you couldn’t get around to it beforehand. Read the transcripts, or listen to tapes, in the order that the groups occurred. It helps your memory and there might be some kind of progression.

“Beginning moderators should work from full guides. Then they can follow the framework of the guide for the report, summing up all comments across groups on each topic or issue area. They can condense comments and select a few choice quotes to represent the spread of responses. Make a micro-summary and micro-conclusion for each topic area before putting it all together for the big section on final conclusions and recommendations. Make sure you follow any reporting or format guidelines that the client may set. These make actually shape the way you analyze the findings.

“I suggest new moderator-consultants avoid being entirely on their own in the analysis process. The best people for bouncing off of ideas are members of your team, mainly clients chatting in the backroom between sessions. Join them, tape them if possible. Maybe you just keep on moderating! Other people to use for trying out ideas are note-takers, if you have one, or research assistants. Moderator colleagues whom you trust are also good sources. Remember that analyzing on your own can be perfectly okay, but keep in mind that two heads are usually better than one.

“If I could only tell someone one thing before analyzing groups, it’s to keep your head until you’ve done all the detail work, then trust your heart to reveal the bigger picture.”

H. Grace Fuller, RIVA Training Institute, Bethesda, Md.

“Analysis is like capturing dreams. First, grab your initial impressions and memories while they are fresh. Write them down, because in the process you remember more than you thought you would. Dreaming comes from the part of the brain that has to do with processing feelings. And it’s more complex feelings that we’re going for in qualitative analysis, rather than just reported behavior. Embrace the ambiguity and uncertainty. Don’t play it safe. Listen to your gut and step out on a limb when reporting. Clients pay us for our insight, not for information they already know.

“Remember, everyone likes a good story. In qualitative analysis and reporting, you are gathering lots of little stories, little bits of the human experience. Simply retell the bigger story, splashed with colorful respondent quotes.”