Great groups demand a group effort

Editor's note: Based in Chattanooga, Tenn., Kathryn Winland is research director at research firm Mozaic Group Inc.

Qualitative research is an excellent tool for refining advertising platforms, messaging and creative executions. At a certain point, however, focus groups and other in-person qualitative methodologies do present some challenges in the creative process. Depending on the size of the stakeholder team and the location, logistics alone can throw a sizable wrench into the team’s ability to fully engage in the research and keep the creative process moving forward at their desired pace (which is usually very fast!). And unless you have the time and budget available to do a half-dozen or more groups, the opportunity for refinement and retesting is quite limited.

Certainly, there are decent workarounds if in-person research is absolutely necessary, but a recent request from PayPal helped me discover another alternative that not only eases logistical pains but also enhances collaboration within the team as well as with the researcher. Perhaps best of all, this approach enables the team to “workshop” ads or other marketing communications without huge additional investments of time or money.

The specific situation and client needs that led to this new approach are as follows:

  • need for iterative process (test initial draft, make modifications based on feedback, test revised execution);
  • accelerated timeline;
  • large and geographically-dispersed team of stakeholders; and
  • need for a widespread, niche audience – which could be difficult and costly to reach in person.

It should also be noted that PayPal is particularly receptive to new techniques; they deserve much credit, in fact, for the success of this approach – a topic I will revisit later.

Refreshing its campaigns

Some months ago, PayPal was refreshing several of its direct marketing campaigns and other communication tools, including its welcome package for new members. As part of the welcome package, PayPal developed a direct marketing campaign, accompanied by a microsite and animated tutorial video. Because successfully activating new members is critical to long-term engagement, the team needed to ensure that the direct marketing piece, as well as these additional tools, were engaging and easy to understand and that they could motivate new members to make even better use of the full potential of PayPal.

My client approached me with a need to do quick-turn research that could accommodate large-team engagement and iterative rounds of testing, modification and retesting. Additionally, we needed to reach a very specific audience and achieve some degree of geographic diversity.

Thus, the virtual workshop was born. It is essentially multiple waves of testing via bulletin board focus groups (BBFG), coupled with group debrief sessions and quick-turn client revisions. This article focuses on a direct marketing campaign but this approach to creative testing works well for all forms of advertising at virtually any stage in development.

Step 1: Recruit one group of respondents for each wave of testing

BBFGs allow for nationwide recruitment, providing geographic diversity and improving the ability to recruit hard-to-reach audiences. In our situation, respondents were reviewing stimulus and responding individually, so we maxed out our sessions at 25 respondents. We recruited for a total of 50 respondents (25 x 2 waves).

Having one panel of 25 respondents made it easy for the team to observe the group on a single board and because respondents were not seeing each other’s posts, the size of the group had no impact on them. If group discussion plays a larger role, it’s best to have smaller groups and set up multiple boards instead.

Step 2: Wave 1 BBFG with Respondent Group 1

It is important to keep the discussion brief and focused. Typically, our sessions are designed to require no more than 30 minutes of the respondents’ time and because BBFG is asynchronous, respondents can complete it at their convenience within a given 24-hour period.

With an engaged client team, collaboration and real-time reaction to results are significant benefits of this approach. In our sessions, clients make notes, ask questions to the moderator and post internal discussions based on respondent feedback.

In the viewing room of a focus group facility, it’s nearly impossible to have real-time discussion without disturbing other observers and missing the action taking place on the other side of the glass. In a virtual workshop, these backroom contributions come at no cost to anyone’s ability to observe. In fact, they not only keep the creative process moving but also aid analysis and help focus the discussion in Wave 2 testing.

Step 3: Team debrief and revisions based on feedback

The research team moderates a group discussion with stakeholders, often by phone, to help guide decisions and next steps, but the client and creative teams are the real players at this stage. They must be prepared to digest and act upon feedback immediately – typically within one or two business days.

The researcher’s job here is to act as a third-party advisor and checkpoint for creative decisions and to make sure results are reflected in revisions to the greatest extent possible.

Step 4: Wave 2 BBFG with Respondent Group 2

Here we add a fresh group of respondents to react to the revised stimulus. Some of the key benefits of fresh respondents are:

  • Fresh respondents have never seen the stimulus, so their reactions will be relatively pure.
  • Wave 1 respondents may have the tendency to zone in on certain areas or create other simplification strategies that could limit their feedback.
  • More respondents equals greater validation of consistent findings across the two waves.

Wave 2 testing follows a similar discussion guide, though exploratory warm-up questions may be streamlined, and more pointed creative feedback questions may be added. Some consistency is ideal in order to highlight the impact of the changes.

Optional: additional waves of testing. Depending on objectives, timing and results of Wave 2, additional waves of testing may make sense. In my experience, if the objectives are focused and everyone works together to make the first two waves as productive as possible, additional waves probably aren’t necessary.

Step 5: Topline summary and debrief

A very simple, focused topline summary is a useful tool for guiding a final team debrief, which is held as soon as one day following the final wave of testing. The summary provides some structure to the conversation and helps everyone make the most of their time during the debrief and make the most of the research.

Using the topline as a living document is also quite helpful to the team. We add to or revise takeaways, make notes of topics that need further discussion and work together to record recommendations and decisions. As a result, the topline summary becomes more than a record of the results and stretches the value of the research even further.

Ultimately, the purpose of the debrief is to make final decisions and possibly identify areas that need more consideration. We often find that the research uncovers higher-order takeaways as well, which we are sure to include in our topline summary so that stakeholders can apply these insights to future campaigns. For researchers, other added benefits of the debrief include closer integration with stakeholders, turning up the volume on the voice of the research and helping guide insight-driven decisions in a collaborative and meaningful way. This has a positive impact on the role of research in the organization and also ensures that the campaign and its stakeholders get the full value of the research (and then some).

Important keys to success

Some of the most important keys to our success with virtual workshops include appropriate resource allocation for moderation and topline analysis, a fully engaged team of stakeholders and getting the stimulus right.

Resource planning. In general, BBFGs require a different approach to moderation and analysis in many ways. The moderator needs to budget at least twice as much (and perhaps even three or four times as much) time as a traditional focus group, depending on the topic and number of respondents. The time trade-off for traditional IDIs, however, favors the BBFG because you can read and react to posts from multiple respondents at once.

Because respondents will be logging in at all times of day, the research team needs to make sure it has moderation support in place during all reasonable hours. In online qualitative, it is critical to establish rapport and encourage high levels of engagement from respondents. Because my firm has moderators in nearly every U.S. time zone, we use two moderators in different time zones for these projects. The two-moderator team expands coverage without being overly burdensome for any one moderator.

One of the often overlooked benefits of BBFG is one that also requires exponentially more time from researchers compared to many other qualitative methods. In a two-hour focus group with six respondents, each respondent will get an average of 15-20 minutes of air time. In a BBFG, however, even one as brief and focused as this virtual workshop, each respondent provides double, triple or more the amount of feedback, which means the entire data pool increases by 2x, 3x or more. This is truly a huge benefit for the research but it requires good resource planning by the research team.

Client engagement. Client engagement really does make or break this methodology. As I mentioned earlier, I credit PayPal for much of the success of this methodology. Here are some of the steps it took that made a big difference:

  • Before the study began, PayPal researchers organized a team briefing to outline objectives, explain how things will work and establish expectations for stakeholder participation.
  • During fieldwork, the entire stakeholder team logged in regularly and made significant efforts to contribute. They planned ahead so they could block out decent amounts of time throughout the day.
  • The client team was very well prepared for the two debrief sessions, with thoughtful comments and clear suggestions on how to move forward. Everyone, from research to marketing to creative, collaborated remarkably well and this allowed the debrief sessions to be as productive as possible.

Stimulus. BBFG is an extremely flexible platform in terms of stimulus, more so than in-person in some cases. Storyboards, mock-ups, videos, screen shots, Web links or even simple messages on a white board can all work well.

The virtual workshop especially shines when stimulus is a bit farther along, such as a first draft of a print ad or, in the case of video, an animatic. This is when the respondent’s ability to independently view and react to each piece of stimulus is a clear advantage and this is often when timelines shrink and opportunities for iterative testing is most valuable.

Of course, whatever form the stimulus takes, the moderator must introduce it accurately and with appropriate caveats as needed; in order for the moderator to not only introduce but also discuss any stimulus, research and creative must be on the same page. The creative team should provide a clear and thorough briefing on what the stimulus is intended to convey and, if applicable, where the focus areas should be. In the case of PayPal, the creative team often seeks input from research in advance, kind of as a pre-test to catch obvious issues but also to make sure the stimulus will work well in the research – a very commendable extra step, indeed.

True collaboration

When researchers and their clients are being pushed harder and harder to deliver quick-turn insights and are being asked to do more with less, we are often left wondering how we can possibly maintain – not to mention improve upon – quality and real research value. The virtual workshop is one example of how true collaboration between client and researcher can help us all not only cope with these demands but actually use them to create something that otherwise might not have come to light. This researcher would argue that the quick-turn virtual workshop truly enhances ad testing regardless of whatever external demands may exist.