A matter of trade-offs

Editor's note: Robert S. Owen is project manager at Sykronix, a Jeffersonville, Pa., research firm.

No one method of conducting research is ideal in all situations. Each method that we might choose for questioning participants in a study has advantages and disadvantages in different situations. Situational variables include constraints on and trade-offs between cost, turnaround time, the level of accuracy that is required of the data, and the appropriateness of the method used to tease out the kind of insights that we need.

For example, personal individual depth interviews provide very rich insights. The length of a pause in answering a question and the expression on a person's face can say more than the words that are transcribed. Interpretation, however, is very subjective, and some interviewers are better than others at eliciting such reactions - suggesting that this richness is also laden with bias. Paper-and-pencil surveys conducted by mail, on the other hand, can be designed to minimize these sorts of biases, but the structured, inflexible methods of mail surveys don't allow for the richness of insights that we can obtain from a personal interview.

What happens when we conduct online surveys? Online survey methods mimic methods that we have used in the past. An online questionnaire is similar in format to a paper-and-pencil mail survey, and an online focus group in a chat room has some similarities to a conference telephone call. The conduct of surveys online, however, presents a different set of costs, of turnaround times, and such. The focus of this article is on outlining some of these features and associated advantages and disadvantages.

Before moving on, it is important to clarify what I mean by "online survey," because the sorts of online surveys that are technologically possible right now are much different from the way that most people conceptualize online surveys. For the sake of this article, "online survey" should be taken to mean the conduct of an html forms-based survey via a current generation, mouse-based Web browser on the Internet or on a company intranet or, much less frequently, interactively through a live chat room or a message board system. This clarification is important because the Internet is not the only form of online communication. The future won't necessarily see us communicating on something like the Internet with mouse-pointing devices that are connected to browsers.

Even now, for instance, we can watch streaming video and can communicate telephone-style through the Internet - but I don't want to confuse this with, say, watching a videotape while talking to a marketing researcher on the telephone. And there are other wild new capabilities that are enabled by Web technology, such as mimicking eye tracking via a mouse pointer - but we haven't yet arrived to any level of acceptance beyond Web technologies that mimic the sorts of tools to which we are already accustomed. The discussion in this article, then, is limited to the context of current generation, standard html forms-based surveys - radio buttons, check boxes, text boxes, and such - and the less frequent use of live chat rooms and message board systems.

Trade-offs in selecting a method

In choosing a particular method, one must consider trade-offs in:

  • ability to answer research objectives;
  • structure or flexibility to be adapted during the conduct of the survey;
  • error and bias that is introduced in taking measures;
  • total turnaround time;
  • monetary cost.

And we consider these trade-offs in comparing basic methods such as:

  • personal interviews (individual or focus group);
  • telephone interviews;
  • mail-based paper-and-pencil surveys;
  • online forms-based surveys.

If we are conducting exploratory research and don't yet even know what questions to ask, then an impersonal questionnaire type of survey would not have the ability to answer our research objectives. An open-ended paper-and-pencil survey might be helpful, but our best choice would be telephone or face-to-face personal interviews because these are adaptive. That is, we can gain insights from the participant during the interview and use these insights to ask probing questions - questions that we might not have known to ask before starting the interview.

On the other hand, if we know that some consumers in our competitive target market are price sensitive and others are brand sensitive, conducting one hour interviews with a sample of 30 prospective buyers will not yield information that will allow us to make predictions of market share in a market composed of a name-brand item and a lower priced generic. Prospective buyers themselves might not even be able to estimate their own price sensitivities, but could rank order a few different products from most to least desirable - a task that could easily be done through html forms on a Web browser. An analysis could quickly be done with an off-the-shelf conjoint analyzer, yielding estimates of market share.

Trade-offs in online research

Ability to answer research objectives

  • Online research via html forms has about the same capabilities as a paper-and-pencil questionnaire. It is well-suited for questionnaires that use anchored scales such as semantic differential or Likert-type scales, and it can accept open-ended, typewritten responses.
  • Online research via a chat room can crudely mimic telephone interviews, either one-on-one or in a group setting. Waiting for individuals to type responses, however, makes this a painful experience for all involved. The introductory exchange, "Hello!" "Hello!" "What is your name?" "My name is Kim" would take less than three seconds face-to-face, but could take a minute online. In other words, the richness in online interactivity is of no comparison to the richness of telephone or face-to-face interviews.

Structure/flexibility

  • Like its paper-and-pencil counterpart, a forms-based questionnaire is usually rigid and structured. Although the questions must be chosen ahead of time, an online questionnaire has two programmed capabilities that add to its flexibility:

    - Branching takes the participant to a different set of questions based on the answer that is given to a prior question. This could be a "simple skip," in which questions are skipped because they would not be relevant to the respondent, or could be "dynamic branching," in which one of many possible sets of questions is presented to the participant depending on the way that s/he responded to a question.

    - Piping integrates responses from a question into later questions. A participant could be asked to type an answer to an open-ended question, and the text of that answer could be incorporated into the wording of the next question.

An online questionnaire, then, can be programmed to be much more dynamic and flexible than a mail survey.

  • Like telephone and face-to-face personal interviews, interactive interviews via chat rooms and message boards can be flexible. However, online methods don't have the same real-time capability and don't allow for as much to be communicated in the interactive exchange.

Error and bias in taking measures

  • Being non-direct, pre-programmed online questionnaires will not suffer from interviewer bias, but we will often lose control over the environment in which the participant completes the survey. In some cases it is to our advantage that the participant is in her/his own home or office environment, as is usually the case with an online survey. On the other hand, we have no control over the distractions that come through these environments. For example, if a household consumer is on a survey panel in which s/he is entered into a drawing for prizes with the completion of every survey, the participant would be motivated to complete a survey once started, no matter the quality of the answers. A paper-and-pencil survey can be set aside to be completed later, but an online survey will be rushed to completion if there is another local demand.
  • An advantage of online surveys is that we can often work with consumer or professional participants who are part of a panel. One reason that university researchers like using students is that repeat participants become adept at using the measurement instruments: people who have previously done, say, a conjoint sorting task now understand how to do it and can provide better (more valid and reliable) answers. Online panel participants, too, will become adept at using the measurement instruments such that we are likely to reduce measurement error with repeated participation. This doesn't mean that they are wise to what answer we want, but rather means that they know how to give us an answer.
  • Interactive online methods will have many of the same interviewer biases that would be found in telephone or personal interviews. They might have additional errors, however, due to the difficulty encountered in the interactive experience. There is only so much that a participant can say when typing at a keyboard, and a burst of ideas can be completely lost across the time that it takes to type a single sentence. Retrieving merely what is at top-of-mind awareness does not give us a complete picture.

Total turnaround time

  • Online questionnaires can be very fast in turnaround time. For a single page questionnaire (no branching or piping), it takes little more time to prepare a survey from an html template than it does to prepare one on a word processor. It takes no more time to upload the survey to a server than it does to send a paper survey down the hall to a laser printer - but now you don't have to walk down the hall to pick it up and don't have to print more copies. If you already have a participant panel in place, you could begin data collection immediately. Online surveys save the data, of course, into a computer file - so you can skip the keypunch step. With a panel in place, you could possibly have data ready to analyze on the same day that you post a survey.
  • Branching or piping questionnaires: A single-page questionnaire can be created in minutes, but one with branching or skips requires a little bit of programming on the server side to create the "if this then go here" procedures. Even still, this might only add a few hours of programming, testing, and mounting to the total turnaround time - if you aren't using a commercial application to prepare surveys (which would be faster). Piping is necessarily customized and is likely to be more complex and require more time - perhaps even weeks of customized programming.
  • Using an online chat facility for a discussion, you also should be able to keep relatively fast turnaround times if you have a panel. You don't have to make travel plans to personally visit different cities, don't have make appointments for the rental of facilities, and, if you have a panel in place, can simplify recruiting and setting up a meeting time for an individual or group of participants. Using a message board panel, you might also be able to obtain responses in a few days.

Monetary cost

  • Single-page online questionnaires cost almost nothing to prepare. If you have technical skills, you can use templates and Notepad on a PC to paste together a single page survey in minutes. The only other tool that you need is an FTP client to make a directory on your server, to upload your questionnaire, and to copy the script (program) that saves the data to a file in your folder. If you don't have these skills and don't have your own server, there are a few organizations that allow you to create your own questionnaire to run on their server and generate analyses at a cost of a few hundred dollars. If you already have a panel, recruiting costs are likely to be low - people participate for the chance to win, say, a watch or cash, or for a small donation to a worthy cause. Of course, you have no copying, postage, or travel costs.
  • Questionnaire branching and piping are more labor intensive, so there is the added cost of these. Contrast this cost, however, with the days lost in travel time for individual depth interviews - during which little is accomplished by a skilled person. If you are a third-party research firm, consider that things like travel costs and facility rental result in billables that contribute absolutely nothing toward internal costs and profits. The time spent to develop a customized survey in-house creates billables that contribute directly to the organization.
  • Online chat rooms and message boards have no variable costs associated with usage and cost nothing to maintain once installed, and these are available as freeware or for a few hundred dollars. Installation does, however, require some technical skills. Once installed, the only cost associated with using these methods would be for recruiting or maintaining a panel.

Advantages and disadvantages

Online research has advantages and disadvantages, as does any other method. Online questionnaire methods are similar in form to paper-and-pencil survey methods, but offer the general advantage of being lower in cost and having potentially faster turnaround time. Online chat rooms cannot provide the same rich quality as individual dept interviews, focus group interviews, or telephone interviews. However, the only cost of doing personal interviews through a chat room or message board is the cost of recruiting. Even with substantially decreased quality, online personal interviews could yield useful insights at the exploratory stages of research.