Qualitatively Speaking: A moderator's guide to working with the creative department

Editor’s note: Michael Carlon is a moderator with Cincinnati-based MarketVision Research. Working out of the Stamford, Conn., office. Michelle Ogren, another MarketVision moderator, contributed content to this article.

I started my research career at Modem Media, one of the first digital marketing agencies. Founded in 1987, Modem is credited with developing the first commercial Web sites and online advertising campaigns. Modem was also the first online agency to have a dedicated research department, of which I am proud to have been a part. We pioneered the use of online surveys, online focus groups, Web site usability testing and online bulletin boards. We did our part to help grow suppliers who were entering this field and we broke a lot of rules. It was a great time to be in research!

While my job was exciting, it became clear, after a while, that there was a tension between those of us who worked in research and those who were responsible for creating the ads/Web sites/technology that paid the agency’s bills (and ultimately my salary). Research was viewed as a department whose job it was to run “beauty pageants” for different ideas that creative teams spent their time and talent bringing to life.

This tension personally came to a head for me when I received a call from the president of the agency, Bob Allen; an imposing figure for those who did not know him well. I was to come to his office immediately. Upon stepping into his office, with my heart beating in my throat, he looked me in the eye and said “I got a call from a creative team who complained to me that you s--- on their creative.”

At the core of this complaint was one statement in a report I wrote summarizing consumer reactions to a Web site we had created years before. The team had just won a project to redesign the site and, as we began this redesign, we thought it wise to solicit input from frequent site visitors on what worked well, what did not work well, etc., to identify areas for optimization. As such, I called out opportunities for improvement, but did so using language that was not as constructive as it should have been.

From this experience, and countless others, I learned a few important lessons that I carry with me to this day:

•  In an agency, creative people and their talent are responsible for delivering the agency’s product. Their concerns are listened to above all others.

•   It is a natural instinct to defend the fruits of one’s labor and developing creative is indeed a labor of love. Evaluating someone’s creative output is akin to talking about their child.

•   No matter the intention of your message, it can sometimes be misinterpreted.

There was a time when I dreaded working with creative teams due to the tension that exists whenever someone is asked to evaluate their output. Over time, though, I became empathetic to the sensitivities of my creative counterparts and started putting myself in their shoes. After all, I view what I do as a creative endeavor and get extremely defensive when someone challenges something I have written.

So, over the years, I have adopted techniques to evaluate creative efforts that are more in line with the sensitivities shared by most people who create things. I hope you will find the following five considerations helpful the next time you are faced with a project that calls for the review of different creative treatments:

Consumers are not copywriters and/or art directors. When you ask consumers to evaluate creative, keep the objectives of the brief in mind as you probe into their reactions. Research about creative is most effective when it identifies whether a creative treatment links back to its intended communication (stated in the brief) and not what people like or dislike about specific elements.

If the creative treatments communicate what they intend to, it is important to document the elements that work. If they are not working, it’s important to provide direction on where the communication may be breaking down. As such, I find it critical that the agency team walk me through the vision behind their work well in advance of reviewing it with consumers. I not only become better educated on the objectives of the research but also find that knowing the intended meaning behind each creative element helps direct my probing during the groups.

Start on a high note. When reviewing general reactions to a creative treatment in a focus group setting, start the discussion with those participants who express positive feelings toward it. Though we never want to bias a discussion, it is important to identify the elements that work and understand why they are important. If you start with the negative, you may not get the cleanest read on what is working. When turning the discussion to those who were less-positive, keep your probing framed in a positive fashion. Instead of asking “What don’t you like about this execution?” ask “How can this execution be improved?” While the difference may appear subtle, creative teams become less defensive about criticism when participants appear to be helping them improve what is not working vs. telling them their “baby is ugly.”

Provide context. Include a section in your interview guide that helps explore the backdrop against which consumers are providing their reactions to the creative. For example, in groups where you are reviewing some online advertising ideas, be sure to address reactions to online advertising in general, what makes a good online ad, what makes a poor ad, situations where one might be more receptive to online advertising, etc. This context may help provide direction to creative teams when optimizing tested executions.

Don’t let creative evaluation be the first research that is done on a particular project. Influence the creative teams you work with to include insight-generating activities early in the creative development process. Helping creative teams experience the target for whom they are creating reduces the chances that a campaign idea might be off the mark. (For more information on how to do this, see my article “Moving from validation to inspiration” in the October 2007 issue of Quirk’s.)

Don’t hang the creative department out to dry. If an idea is totally tanking, the moderator may need to fight for it a bit (in an objective way) in order to a) completely understand the problems that exist and b) show some solidarity to the greater team. It’s more difficult for the creative agency to shoot the messenger if the agency gets the sense that the moderator is in the trenches with them.

Always be tension

There will always be tension between creative teams and researchers, regardless of whether consumer research is viewed as an invaluable part of the creative development process or as a necessary evil. The extent to which this tension is healthy is entirely up to the personalities on both sides of this aisle. Researchers looking to improve their relationships with creative teams must keep in mind that there are constructive and non-constructive ways to evaluate creative. It is my hope that more of us take the former path rather than the latter.