Editor’s note: Bill MacElroy is president of Socratic Technologies, Inc., a San Francisco research firm.

The following questions are the ones that I hear most frequently from people considering using the World Wide Web and/or the Internet for conducting marketing research.

Q1: Are the results that you get using on-line research techniques projectable?

A: The question of projectability always needs to be examined in the context of the research being done. Statistical projectability says that if you take a random sample of sufficient size within a known population (e.g., that everyone in the population has an equal chance of participating), then the results are indicative of what you would get if you polled the entire population, give or take some level of error. The real trick here is to answer the question "What population are you polling?" If you are attempting to determine the opinions of on-line users of a particular site, or users who register products on-line, and you randomly recruit them to participate, then yes, your results will be projectable to that population. If you are attempting to determine reaction to a new concept for a salty snack, however, using on-line research probably won’t map to the total population of consumers of snack products. The universe of on-line users (both business-to-business and consumer) has grown to the point where it should be considered a sub-group of such importance that it is worthy of study in and of itself – without the questionable practice of attempting to project these findings back to the more general, off-line population.

Q2: How do you recruit people for these surveys?

A: Recruitment for on-line studies is done in one of several "ethical" ways. The best results for recruitment are achieved by contacting people who are "known" participants in your own on-line activity. These can be product registrants, people who have signed up for your on-line service, guest registrants on your Web site, etc. It is usually recommended that you include the request to contact registrants for possible follow-up research as a part of any on-line registration process.

There are several nationally known on-line databases (Socratic Forum, Greenfield Online, Cyber Dialogue, etc.) which can be sources of potential respondents. These databases consist of people who have volunteered to take part in on-line marketing research projects and who have been screened for on-line activities, as well as demographic, firmographic, and technographic characteristics. Unlike some traditional panels, these databases are usually not over-polled. Incentives are set for each survey based on historical response rates for similar topics and questionnaire length.

Once the source of potential respondents has been identified, several methods of recruitment may be successful. For extremely sensitive or complex surveys, some studies are pre-recruited by phone. This can be somewhat expensive and cumbersome, particularly if passwords are required. Also, some respondents have difficulty in writing down complex Internet addresses (URLs such as www.survey.com/bin/auto_test~1), so it’s best to keep them as simple as possible.

The best way we have found, however, is to use an e-mail invitation that has some very specific components. In order to be successful, the e-mail invitation should include:

1. A subject line that indicates the topic (e.g., "Survey Invitation for on-line Travel Purchasers")

2. Where the e-mail address was found (e.g., "As a recent registrant at our Web site…")

3. Who is conducting the research (e.g., "XYZ Company has been commissioned by [Client Name optional] to perform this research). If the study must be blind to avoid biasing the answers, we often offer to reveal the name of the client at the end of the survey, or by follow-up e-mail if so requested. Note: If what you are testing is so confidential that the client can never be revealed, you may want to reconsider using the Web as the delivery mode.

4. A brief description of the topic (e.g., "We are interested in the opinions of those who have booked hotel reservations using an on-line travel service…")

5. The approximate time required (e.g., "This survey should take no more than eight minutes of your time…")

6. A description of the incentive. Note: Very few people do anything on-line without some type of compensation or honorarium! (e.g., "The first 100 people to complete this survey will receive…")

7. The survey address. Note: Most modern browsers will automatically detect a valid Web address and turn it into a hot link – allowing people to immediately go to the survey site.

8. Valid contact information (e.g., "If you have any questions about this survey, or if you experience any technical difficulties, please contact the Project Director Brian Smith at 800- 555-1234 or e-mail him at brian.smith@researchco.com)

Q3: What is the difference between an e-mail recruitment and all the junk mail (spam) that people get?

A: One way to recruit people for on-line studies which appears easy but is considered extremely offensive, is to send out mass e-mails to all sorts of people with no regard to the relationship they may have with the sponsor of the research. Not only is this considered very poor "netiquette," it also doesn’t work! In an academic study run in 1995, controlled tests of response rates between paper and pencil and unsolicited, general e-mail surveys showed that even paper and pencil had a higher rate of return than spam e-mail surveys. In addition, the number of hate-mail responses was five times the number normally seen in other forms of research. Furthermore, the quality of data from the completed unsolicited e-mail surveys was terrible – many more missing answers, obvious sabotage, and a high number of unrealistic outliers demonstrate that SPAM DOESN’T PAY!

Q4: What are "cookies?" Are these ethical? Can’t people get around these?

Cookies have been the source of a great deal of concern and alarm in the press. In simple terms, a cookie is an incredibly small bit of information that is stored in a user’s browser. These files do not "escape" from the browser and can’t affect the computer itself. Cookies are used to store information about what you have done on a Web site. Any sites that need to "recognize" you without a password (such as a customized news reader, MyYahoo!, or other personalized sites) do so by storing the key to your profile as a cookie. Cookies can be used in research to determine where a respondent has left off in a survey if they wish to pick it back up later. Cookies can also be used in conjunction with unique URL addresses and other technologies to help enforce quota controls and multiple-survey submissions.

Although most new browsers allow users to either turn off the ability to write a cookie, or alert the user when a cookie is being written, most people who take advantage of these options lose a tremendous amount of Web functionality, or are prompted to accept or reject a cookie constantly. Most people quickly decide that the cookie writing functionality is worth any intrusive aspects.

The only thing that could prove unethical with regard to the use of cookies has to do with spam. Some new technologies can read what sites you have visited and add you to a contact list that usually involves an unsolicited e-mail. Another, less intrusive technique involves using this same technology to create customized banner ads for your "surfing profile." For example, if you visit a lot of travel sites, you might begin to see more banner ads for travel specials and on-line agencies.

Q5: How can you make sure that the person you invite is the one taking the interview?

A: Interestingly enough, there are more technologies to make sure that you are dealing with the correct on-line respondent than in many traditional techniques such as paper and pencil. One of the best ways is to generate a unique Internet address for each invitee. This can be done by using a random generator to tack on a short address to the main survey URL. If, for example, we generate the random string "xt)5~P," we can append this to the URL address in the individual invitation to participate (either through the mail or using e-mail). The resulting address, www.survey.smart. com/sbs/exqm.xt)5~P, is unique to the recipient of the invitation. In conjunction with cookies and other technologies, this method allows respondents to click on the hot link and go to the survey site and either start the survey from the beginning or pick up where he or she left off. As the respondent attempts to enter the survey, the pre-determined address code is compared to a database. If the survey code is valid and has not been submitted previously, the survey admits the respondent. This system allows the respondent to complete one survey from any number of machines (either at home or work), unlike systems that rely solely on cookies for screening purposes.

More recent developments from major browser producers allow for electronic signatures from individuals who have entered a password for access to the Internet. In the future, these "signed" documents will help certify who has submitted a completed survey.

Q6: What kinds of security issues do you have to worry about when conducting on-line interviews?

A: Actually, very few security problems have ever been documented that didn’t involve people voluntarily giving credit card numbers or other sensitive information to unscrupulous operators. Most security issues that have been raised with regard to survey technology have to do with electronic eavesdropping or "data clipping." Clipping can occur when all the data from a survey is transmitted at once as a block of information (such as when you complete an entire registration form and press "submit" at the end). Although EXTREMELY rare, someone with the right equipment could technically capture the data from your respondent’s completed survey, and if they knew the site address, could reconstruct the answers to the questions. Obviously, this is a lot of trouble to go through for the answer to one questionnaire.

In the newer, more interactive forms of Web surveying, the data is submitted as smaller blocks (particularly whenever the database needs to test a set of answers for skipping instructions, randomization, or other logical functions). Because the Internet sends each batch of information through an infinite number of routes, it is now almost impossible to get anything more than one or two disassociated answers. In addition, because the survey itself is interactive, it is almost impossible to figure out which questions go with the answer data you’ve clipped. Bottom line: Interactive surveys on the Internet are MUCH more secure than telephone or paper and pencil methods.

Q7: What are the various options for conducting on-line research and when do you use one versus another?

A: At this time there are six types of on-line surveying being done. These are:

  • E-mail and bulletin board surveys: Usually best with internal audiences (employee groups, distribution partners, beta testers, etc.). Advantages: Very fast. Very cheap. Limits: Best with fewer than 12, close-ended questions. Not all e-mail/bulletin board readers support graphics. No skip patterns, error checking, randomization, or other logic-based control techniques are available.
  • HTML form surveys: Use form-based data entry process and submit all survey answers upon completion of data entry. Advantages: fast, fairly inexpensive. Limits: best with shorter, more straightforward surveys. No skip patterns, error checking, randomization, or other logic-based control techniques.
  • Fixed-format interactive surveys: provide interactive testing using a standard format for question construction. Most examples of this type of survey tool are stand-alone, PC-based software packages; some, however, can be designed and implemented on-line at a survey Web site. Advantages: fast; full use of skipping, piping, randomization, and other logic-based controls; less expensive than custom programming; built-in QA functions. Limits: less flexibility in formatting. Usually only one question per Web page or some other type of highly regulated question format.
  • Custom interactive surveys: Advantages: provide all the benefits of fixed format interactive surveys with the added benefit of greater flexibility in Web page design and better presentation of graphic stimuli. Allow multiple question blocks. Limits: Much more expensive. Take longer to program and do QA.
  • Downloadable surveys: Are programs that are designed to run on the respondent’s computer, rather than on the host Web site. Advantages: larger number of fancier, Windows-based controls. More exotic testing is available (e.g., complex trade-off exercises, sophisticated allocations, and so forth). Limits: expensive to program. Long download times for even short surveys (10+ minutes). Difficult to explain/obtain respondent compliance in returning data file.
  • IRC/chat functions: Interactive chat, used in qualitative research for both on-line focus groups and one-on-one in-depth interviewing. Advantages: faster than real focus groups. No geographic barriers. Limits: somewhat chaotic – relies very heavily on the skill of moderator to provide control. Tyranny of the best typists may skew results. Less interaction between participants than in physical groups. Difficulty in getting specific individuals together at one time with all telecommunications functions working properly at all sites.

Q8: Can I run these surveys on my own equipment/server?

A: Right now, several of these scenarios can be hosted by the client’s own server. Once the researcher has created an e-mail or HTML form-based survey, these can be put up on the client’s internal server. Other more sophisticated forms of Web-based surveying (fixed format and custom interactive surveys) are currently being hosted only on the producers’ sites.

Q9: What kind of incentives do you find effective for on-line studies?

A: Many different types of incentives are being offered to potential respondents in an attempt to increase participation rates. These fall into several different categories. The most effective incentives (from a historical perspective) have been drawings for cash prizes of $500 or more, or for computer or office peripherals (e.g., printers, PDAs, telephones, etc.) For very hard-to-reach respondents, individual incentives of airline bonus miles or cash awards of up to $50 have proved extremely effective. However, individual cash incentives of $100 have proved no more effective than $50 cash premiums in attracting difficult-to-reach respondents.

Occasionally, someone asks about states that don’t allow "lottery-based" incentives. What we have determined so far is that Web-based surveys are somewhat different than other forms of surveying in that all transactions take place at the site where the survey is hosted (the individual survey-takers are only connecting to the Internet at their site – not doing the survey). So far, if the server is in a state that allows these types of incentives, then they are permissible. Even though laws limit the use of the telephone to solicit or offer lotteries or other illegal forms of gambling in some states, the Internet has been determined to be different and distinct from this interpretation of "telephonic technology." At this point it looks doubtful if the individual state governments will be able to successfully regulate this type of e-commerce. Be careful, though, if you plan to pre-recruit respondents by telephone or to download surveys that run on the respondent’s PC – these are probably covered under the existing laws. Also remember that the Internet rules are "in play" and may change at any time.

Q10: How long does it take to get an on-line survey done from start to finish?

A: This, of course, varies according to the length of survey and methodology used, but here are some average times from final questionnaire to quantitative topline data:

E-mail: three hours to three days

HTML form: three days to 10 days

Fixed format interactive: three days to 10 days

Custom interactive: seven days to 14 days (includes three days of custom programming/QA)

Downloadable: seven days to 20 days (includes three days of custom programming/QA).