More than Doom & gloom

Editor's note: Carla Sarett, Ph.D., is a research consultant with Chilton Research Services, Radnor, Pa. This article is adapted from a presentation made to the Advertising Research Foundation conference on "New Frontiers in Market Research" in May.

There is a lot of excitement about the new data collection possibilities posed by the on-line world. Re-searchers are captivated by the idea of collecting data quickly and cheaply from respondents in this new environment. Companies are rushing to set up their own on-line panels, focus groups and Internet/Web surveys. Before we get too far along, it's important to sort out what we are learning from these on-line experiments and what kinds of data are appropriate to collect.

Chilton Research Services organized an on-line panel in 1995, from June to December, through a proprietary bulletin board operated by the company. We collected data on the general area of technology and media from teens (aged 11 to 18, living at home). The aim of the on-line panel was to collect qualitative data which were nationally projectable. We recognized that our panel would be comprised of teens of a special kind, that is, teens who owned computers with modems. Still, if our panel of teens were representative of modem-owning teens in general, and could provide interesting qualitative data, it would meet our criteria of success.

Creating an on-line panel

A look at how we put this on-line panel together will give you a sense of the difficulties involved. At the start of this project, we were fortunate enough to have a national sample of households with teens drawn from our omnibus surveys. We had, also fortunately, conducted an interview with an adult in the household, so we had household data going into the study. From this sample, we conducted a telephone survey with 750 teenagers and collected data on their media and technology usage, along with general attitudes and demographics.

We could therefore identify teens with access to computers with modems, either at home or at school. This gave us our eligible base; 350 teenagers qualified to be part of our panel, and 138 of these agreed, when asked to participate after completing the telephone survey. We then sent invitation letters, along with legal disclaimers, to these qualified teens, giving them a toll-free number to dial. We had decided not to implement this panel through the Internet because we wanted to make it accessible to teenagers with older computers. We hinted at future goodies but gave no incentive.

Because the legal issues involved in collecting data on-line from panels of kids and teens were extensive, we enlisted the help of our legal department. We were concerned that we not, as researchers, be responsible for any pornography on the on-line panel, so we limited interactive chats and had a system operator reviewing messages and E-mail. We made it clear to teens that their comments were not private.

We also put in place validation checks to ensure that the teens on-line were the same teens we had spoken to over the telephone. This meant that our sample could not snowball into a much larger, but unidentifiable, group. The proper identification of panel participants was key, because we wanted to have qualitative data that could be linked, at any moment, to the survey data we had collected on usage/attitudes. Also, from the survey data, we knew on-line teens did not express different attitudes from other teens or use other media, apart from computers, very differently. For example, they were not more inclined to believe computers were important for academic success; nor were they less sociable than other teens.

On-line panels have their own dynamics

Our initial results were encouraging: over 50 teens signed on to the board within a month and the discussions were lively. Teens posted messages constantly through a bulletin board-style service, and more teens, with additional reminder notes from us, began to join. However, after a few months, we noticed a sharp drop-off.
This drop-off in itself was not alarming; all panels need refreshing. But this differed from normal panel attrition. Here, it was the dynamics of the on-line panel itself that made teens drop: certain teens were "discriminated against" because they didn't fit in. Typically, panel designs don't rely on respondents getting along with one another, but because we had created an on-line community, our panel design demanded that respondents communicate with each other. This was very different from a focus group. These teens had to live with one another on-line.

We expected from the outset that on-line/Internet panels would favor respondents who enjoyed writing. However, we underestimated the effect of assertiveness. Girls, initially enthusiastic, began to lose interest because boys were more assertive verbally, often making nasty comments and making cyber life less fun for girls. By October, we had a small cohort of articulate, opinionated teens, mostly boys. We had lost many girls and also those teenagers who were more timid.

With this change, the diversity of opinions which initially existed on-line withered. So, after all the care we took to create a nationally projectable group, we ended up with a select group instead. We then had to work to bring back those who had left by sending them private messages by E-mail or traditional mail. When looking at data derived from Usenet groups and Web sites, it is important to keep this bias in mind.

Quantitative data collection questionable

Because we recruited our panel by telephone, we knew who was on-line and who had left. It was this knowledge that made quantitative research seem undesirable. However, many companies conducting Web research don't have data about their on-line samples. While the large respondent pools visiting Web sites are tempting, the use of Web surveys as a substitute for traditional primary research seems misguided. If companies want to measure response to a Web site, then using an on-line survey is fine, similar to an insert card in magazines. However, to regard on-line surveys as an improvement on insert cards seems premature.

The promise of "invisible" data collection

Our goals were never quantitative: we wanted to obtain qualitative data through a representative on-line panel. We found that while you can collect great qualitative data on-line, on-line data are no substitute for focus group data, and they require just as much time and effort to obtain.

On-line data collection has to be supervised. Teens wanted us to provide a grown-up leader. We at first intensively supervised the group bulletin boards through an in-house system operator (a "sysop") who questioned the group and admonished teens when language became inappropriate. However, we felt that this procedure was too intrusive and instead opted for an open environment; we allowed teens to guide their own on-line discussions, intervening only when they misbehaved.

This was a mistake. Teens, it turned out, did not enjoy an unsupervised on-line experience. They needed us to guide them, stimulate discussions and lead them to greater insights. In other words, they needed a moderator.

The promise of an invisible, seamless data collection process - one in which researchers more or less "overhear" rather than "gather" data - is just a promise. If you want to get data on marketing issues, you need to take an active role on-line just as you do in a focus group or in an in-depth interview.

What can you learn: the case of games and gaming

Even with all of the caveats discussed above, on-line data collection offers some genuine advantages.
Computer games and on-line gaming was a perfect kind of market to explore through our panel. We had highly sophisticated kids who had modems, liked using computers and liked expressing their opinions to one another. If we wanted to learn about how teens responded to CD-ROM games, these teens were excellent candidates. Furthermore, we knew at the outset that teens' involvement in games was intense, so it seemed on-line panels could shed light on the nature of that experience.

Because our on-line panel was a kind of mini-community, the teens started to treat one another as friends with whom they would exchange advice and/or recommendations. Therefore, we could see how kids developed word-of-mouth about games and gaming. Teens liked to recommend not only specific games, but also branches of games like MYST. Teens asked pals on-line if they had heard of specific games, like Full Throttle and ROTT, and what they thought of these games. We could monitor and "overhear" these recommendations.
Teens were savvy about gaming, and they were also pretty thrifty. When one teen asked if he should buy Ultimate Doom, he was quickly warned: "I've played Ultimate Doom, and it's not really all it's cracked up to be. Why spend $50 for the game when you can get a patch off the Internet for free? Anyway, the fourth episode is OK, it just doesn't really keep the Doom spirit."

DOOM and MYST still "ruled" and "rocked" during our study. The cult-like involvement surrounding these games, along with their violent imagery, made them irresistible. When teens solicited help with a gaming problem from their on-line friends, other teens rushed to the rescue. Teens use games, we saw, to test their abilities, and they want to be challenged. They don't want the game to be uncomplicated. They want it to be tangled and complex; the more branches, the better. It's hard to imagine another medium - even books - competing with the involvement that teens have in computer games. As one of our teens, who called himself Wolverine, put it, "One ending is pretty boring if you ask me."

Learning the vocabulary of games

Talking about the game is part of playing the game, along with learning its intricate vocabulary and rules. Our on-line panel's discussion quickly became almost indecipherable to anyone not familiar with gaming lingo. Take the following conversation thread:

"Man, I was playing DOOM2 and I went into this room that had like 15 Barons of Hell and a CyberDemon. By mistake, I fired my combat shotgun and in like 5 seconds, I was down man."

To which another teen answers:

"What you have to (do) with that many big guys is put like a rocket shot into the CyberDemon, then close the door and listen until the dying screams stop."

A teen calling himself Highlander fires back:

"Good point. But I've got a few better ideas. Pull out the Plasma Rifle and spread them guts around the room. #2 if you've got a LOT (200%) life, a few BFG shots would do nicely."

We also saw how teens categorize games and gaming. Gaming was a matter of involving oneself in a close-knit group focused on winning the game. Certain games worked for teens "because you had to use your head" while for others, the "only object is to kill everything in sight and get to the exit."

Teens liked the fighting games and they definitely enjoy the visual aspects of aggression. As one of our panelists put it: "One time a CyberDemon chased me through a door and I ran at pack of Imps and one of the cyber missiles hit them and it was raining gore the rest of the day. It was cool." Or another teen, known as NJDFan, who recommended ROTT to our system operator because "it's more gory than you could ever believe." Guts and gore are definitely not beside the point - but, then again, they are not the only point.

Good games let teens feel that they are really fighting enemies - while they are crouched over their computer. Boys crowed about "mowing 'em down" or "getting right in their faces and KABOOM with da bazooka." Reading over these postings, and knowing what we did about these academically talented, computer-literate kids, we found it almost comical to hear them expressing all this aggression. Naturally, that is part of the point of what games mean to teens.

Understanding the gaming experience as mastery

Playing these demanding, violent games in this protected mini-community of like-minded players was key to the gaming experience - winning through brains and knowledge about the game, rather than strength; having a small coterie of fellow teen players with whom to strategize and achieve victory; acquiring a closed vocabulary system to communicate with your gaming peers.

Good games, like DOOM and MYST, help teens achieve these goals. They aren't just about guns and shooting, they are about mastering the world. They allow kids to feel like commanders and captains, leaders and explorers. They allow kids to feel that they have secrets that others (outside the game) don't have: secret maps, secret vocabularies, secret rules. Whereas other teens may say, "been there, done that," our panel of gaming teens boast, "been there, beat that." Games that don't "take long to beat" are definitely not good games.

Our survey data had suggested to us that teens are experiencing grave anxieties about their future and sensing that the world is a dangerous place. Teens see the world today as filled with guns, economic uncertainty and family instability. Teens need to explore boundaries and find safe areas in which to test themselves and feel cool. Successful games manage to achieve this for teens. Games are one way in which teens test this danger. They can win - but winning can't be too easy, and there can't be just one way to win.

The promise of on-line panels

As researchers, we are constantly trying to identify underlying consumer needs. We know that consumers continually turn things on their heads and reinvent the rules. From coffee to beer to Star Trek to DOOM, markets and consumers pursue paths that are often hard to predict. One of most gratifying aspects of qualitative research is that it sheds light on that creative part of the process.

Here, we peered into the world of teens and games. Games may start out as a mass medium which is marketed broadly, but they are used by teens to express themselves to small, exclusive groups. On-line panels are ideal data collection venues for this kind of experience. All of us continually create these groups through fashion, music and media. In the future, we feel that on-line research will be a significant part of our effort to connect with these groups and to gain insight into their shared interests and perspectives.