Editor’s note: Bill MacElroy is president of Socratic Technologies, Inc., a San Francisco-based research firm.

As the demographics of the Web inch closer to resembling those of the general population, the use of new online forms of marketing research is growing rapidly. With it grows the sophistication of the techniques used to conduct these studies. Unlike the early ’90s, when the array of online survey development tools was extremely limited, many more options are now available. Researchers are now less constrained by the shortcomings of the medium and can once again concentrate on the higher value-added functions: research design and analysis of results. Better matches between the research objectives, technology and budget are now available.

This article will describe some of the common forms of both quantitative and qualitative online research being used for commercial applications and will discuss specific trade-offs between research design flexibility, degree of control, relative cost and speed.

In general, there are seven current methods/technologies for conducting online research projects. These range from the most basic, least costly methods (e.g., text e-mail) through highly sophisticated, and relatively expensive forms:

  • e-mail (text);
  • bulletin boards;
  • Web HTML;
  • Web fixed-form interactive;
  • Web customized interactive;
  • downloadable surveys;
  • Web-moderated interviewing: chat interviewing and other discussion formats (qualitative).

E-mail (text)

One of the earliest methods for conducting surveys over the Internet or over a company’s internal system (intranet) is the simple text-based e-mail survey. These surveys can be generally thought of as online paper-and-pencil surveys.

Like a traditional mail survey, there are few, if any, interactive controls or logic testing. The whole survey can be seen by the respondent at once; there is no ability to screen out unqualified or security-risk respondents during the survey. Respondents can change their answers at any point in the survey and can actually change the wording of the questions to reflect what they feel is a "better question."

Early forms of text e-mail surveys had no automated data accumulation. This meant that the researcher had to receive the e-mails, print them out and re-enter the data into standard database format. Newer software allows the researcher to encode a survey for semi-automatic data accumulation as the return e-mail is received.

On the plus side, e-mail surveys are very easy, cheap and very fast. Almost anyone can do e-mail surveys with little or no formal training. The costs are very low and the return speed is very fast. (In some cases, we’ve seen e-mail surveys completed and returned within 12 minutes of the initial sending.)

The ideal use of this form of online research seems to be for internal corporate surveys. This is primarily because these audiences tend to behave better (e.g., follow instructions, fill-in relevant information, skip questions that don’t apply, not change questions, etc.) than external audiences. Even with these groups, an e-mail survey should be short (i.e., less than 20 questions). When respondents have been asked about likelihood to participate in future surveys using various online techniques, e-mail has the lowest rate of future participation interest. When probed for reasons as to why future participation likelihood is low, "too boring" is a frequent comment attributed to e-mail surveys.

Bulletin boards

Another online research form that falls into the "moderately easy, fast and inexpensive" category is bulletin board research. Bulletin board conferences are useful for a "modified Delphi method" for collecting responses over time. The technique involves inviting people to a specific Web site where a discussion topic is posted. As people respond to the question(s), others can eventually see what others have written and respond to the original responses. In this way, the thread of the conversation weaves back and forth like a slow-motion focus group.

Putting up a bulletin board is not difficult, but does take more skill than creating an e-mail survey. Unlike other forms, there is no automated data accumulation of which we are aware. Consequently, the cost of this technique is somewhat higher than e-mail - particularly due to the time needed to handle the comment transcripts and code responses for any quantitative appraisal. Although graphics and other visual stimuli can be incorporated in the original question, the format of the reply system is fixed and not very flexible.

This technology is very good when a panel of experts or beta testers need to post quick reactions or discuss impressions with others. The method combines elements of both quantitative and qualitative techniques and the conversational transcript provides extremely rich data. Interestingly, respondents taking on-line surveys of any type tend to give much longer verbatim comments in response to open-ended questions than traditional method survey takers.

Web HTML

The most common form of online surveying is the flat HTML survey form. By our estimates, almost 80 percent of all survey data being collected online is being done using these forms. Almost everyone who has registered at a Web site is familiar with these types of surveys. Characteristically, these surveys take the shape of a long, single page on which the respondent clicks buttons and boxes, fills in text boxes, and eventually submits the information all at once.

The HTML form survey requires additional programming skill to capture the data upon its submission. This is usually done using CGI (a scripting-language that reads the data into a database when it is received). As an alternative, some new hybrid software packages can be used to both generate the HTML code and automate the accumulation of data upon its submission. Either of these extra steps adds to the time and cost needed to field a survey using this technique.

Although these surveys have no true interactive controls (no true skipping, no way to limit answer changes, no real-time error checking, etc.) these limitations are offset somewhat by the tremendous flexibility of design that can be achieved. Graphics, audio and video clips, animation and many other multimedia forms of stimuli can be used. For quick studies that don’t require complex logic, the HTML form-based survey can be a faster, lower-cost alternative to more sophisticated techniques.

We find that these types of surveys are very good for user registration and profiling studies and/or for studies requiring exhibits, but no complex logic or controls.

Web fixed-form interactive authoring tools

Another new form of online research is being driven by survey authoring tools. Many of these tools have been developed from previous generations of software used to conduct computer-assisted telephone interviews (CATI) or disk-by-mail (DBM) studies. They have been adapted to "play" questions on the Web the same way they would play for an interviewer at a tele-polling station.

The big innovation is the option of allowing the individual researcher to construct highly sophisticated studies for the online environment. These software packages put many of the sophisticated controls that have been available for phone studies since the late ’70s directly into the researchers’ hands. Most of these tools exist as packaged software programs that the researcher uses on his or her own PC. As an alternative, several interesting new Web sites have emerged which allow the author to design the research online without the need for loading the design software.

The reason why these tools are referred to as "fixed format" is that they typically limit the range of options in which the survey can be displayed. Some packages have a very specific way in which each type of question must be designed and/or displayed. Others are more flexible in terms of layout, but limit the number of questions that can be displayed on each Web page. Either of these alternatives limits the design flexibility of these tools. Software in this category ranges from very easy to use (but not very powerful) to extremely powerful (but not easy to use). For now, the individual researcher will have to choose between these two problematic alternatives.

Using these tools to create the survey does not always mean that the researcher can control the whole research process. Few of these tools allow the researcher to "host" the survey they’ve created on their own internal servers (the equipment used to directly connect to the Internet). Most software firms will offer low-cost hosting services on their own servers, but won’t (usually can’t) allow self-hosting. Of those software manufacturers that do allow self-hosting options, the cost is usually quite high (in the range of $30,000 to $60,000). Unless the researcher is doing 10 to 20 medium-sized surveys per year, the economics of self-hosting are usually prohibitive.

Web customized interactive programming

The most powerful and flexible of all online surveying options are those that involve the custom programming skills of highly skilled technical people. They also tend to be the most expensive. Like the fixed-form tools, custom programming provides all of the modern technical controls (screening, skip-patterns, logic, error-checking, etc.), but also offers many other tricks and options that allow the researcher the highest level of flexibility currently available for design and functionality.

Flexibility in layout is one of the chief benefits of this option. Unlike fixed-form tools, question/response styles, backgrounds, graphics, etc., can be more flexibly chosen to best meet the aesthetic wishes of the researcher. There are also benefits associated with extended functionality such as running a live Web site for evaluation within the survey frames, running procedures such as file downloads during the survey or creating dynamic and more descriptive navigation messages.

Increased flexibility and control, however, come at the expense of time and cost. Custom designed surveys take about twice as long to program as those using fixed-form tools and can be twice as expensive as surveys authored using fixed-form tools. Time and cost can also be affected by the degree to which customizations yield complex data structures which must usually be manually tested. Hosting custom programmed surveys is almost always done on the programmers’ servers.

Downloadable surveys

Another online survey method attracting attention are surveys that are downloaded from the Web and run on previously installed software provided by the researcher. This shifts the computing tasks from the online server to the respondent’s PC. Once pre-loaded, the survey software can then read much smaller files that the respondent downloads from the Internet. The result is surveys that run in a very similar manner to the fixed-form interactive surveys. Once the survey has "played" on the respondent’s PC, a data file is created which can then be uploaded the next time the Internet is accessed.

Another variant of this methodology are completely custom programmed surveys that can be downloaded for single-use surveys. This technique gives the research the most control and flexibility of any option mentioned in this article – primarily because the survey can take full advantage of controls, logic and other functions available with the PC’s operating system that might not be supported on the Web. Examples of this type of functionality might include time exposures or live color calibration.

All downloadable studies tend to be more costly and time-intensive than other forms of online research. In some instances, they can also require a greater level of respondent sophistication in order to install software and correctly handle the data upload process. If the survey software must be downloaded in order to take the survey, the time requirement (from 20 minutes to two hours) may discourage some respondents. Allowing the respondent to complete the survey offline (rather than collecting the data live) and then re-establish the data upload connection can also lead to delays and lost surveys.

Typically, we see this form of online research being used with panels or pre-recruited groups who regularly communicate with the survey organizer.

Web moderated interviewing: chat interviews and other discussion formats (qualitative)

The last form of research that is taking place online are the qualitative, real-time chat interviews. Although some people refer to these as online focus groups, we believe that they are significantly different in style and output to warrant a different classification.

In these chat interviews, the logic and control mechanisms are supplied by a highly skilled human moderator. People enter the interviewing chat session and then type the answers to questions posed by the moderator. While the results from traditional focus groups can be highly influenced by the skill of the moderator, these online chat sessions are doubly tricky. Just as the traditional moderator must control the overly enthusiastic participant, the on-line moderator must control the "tyranny of the fastest typist."

The key benefits of chat interviews are highly related to the non-physical nature of the medium. By conducting these discussions online, respondents from far-flung regions can be brought quickly together with no facility or travel expense. Session moderation fees are generally higher than those for traditional focus groups (due to the increased technical skill requirements), this is usually offset, however, by the previously mentioned cost savings. Pre-recruiting non-panel participants to any form of online study remains as expensive as recruitment for traditional studies.

Broader range

New methods and technologies are giving researchers a much broader range of options for conducting research online. Expense and timing considerations can be better matched with need for controls and flexibility in questionnaire design issues. At last, the objectives of the research can be the key focus of the activity without the limitations of the medium playing a dominant role in the assessment of options.