Editor’s note: Larry Herman is vice president at BAIGlobal, Inc., a Tarrytown, N.Y., full-service market research firm.

When quantitative research is presented as evidence in the courtroom, it comes under a different and more meticulous scrutiny than when it is used in making business and marketing decisions. Research for litigation is literally cross-examined on the witness stand.

Opposing lawyers examine every detail involving the execution of the project with the explicit intention of questioning your research and undermining the findings. Even the smallest mistake in the research process can destroy a study’s validity. And, once a survey has been discredited, the expert testimony and sometimes the case can be too!

Given such a hostile environment, researchers who enter this arena must follow precise rules and procedures to eliminate all error and produce clear, defensible results. Having managed hundreds of legal surveys over the past 15 years, and having conducted my own study at my firm, BAIGlobal, of lawyers who commission such research, I offer the following insights, cautions and rules-of-the-road for those who venture into this treacherous niche of the research marketplace.

A new and unfamiliar cast of characters

When embarking on a legal research assignment it’s important to know the wide range of players involved in using your research, both behind the scenes and in the courtroom.

First are the corporation’s attorneys and management executives who articulate the legal claims to be resolved. Legal disputes usually involve business issues such as:

  • intellectual property disputes regarding trademark or copyright infringements;
  • claims substantiation for advertising and marketing materials;
  • misleading advertising involving consumer confusion issues.

If a trial is anticipated, each side will hire their own outside trial lawyers to develop and argue their case in court. These lawyers, in turn, hire their own expert witnesses, usually well-known marketing experts or business school professors, to provide an objective opinion about the subject matter in question. The expert witness custom designs a research study for the trial and, finally, a market research firm with legal research expertise is hired to execute the study.

To make the best case possible, both the expert witness and the market research firm must be legal research specialists:

  • The expert witness is the design specialist and "testifying" professional. He or she must create an unbiased methodology, questionnaire, and sample design for the study, so that the research results are viewed as truly objective evidence in court. Then he or she must be able to testify and explain the survey results articulately and concisely on the witness stand, holding up graciously under cross-examination by opposing counsel.
  • The person on staff at the research firm is the expert involved with the execution of the project, with extensive experience directing and coordinating all aspects of legal research. He or she must ensure that the data is collected and processed without any question regarding its accuracy. Although the market research firm’s legal specialist rarely appears on the witness stand, his or her work will be subject to meticulous examination by opposing counsel as well.

Hostile environment

The clients in a legal assignment have an attitude towards research that is radically different than business clients in a typical research project. Trial lawyers tend to be suspicious and distrustful of research, viewing surveys as a necessary evil. They know that opposing counsel will be looking to discredit their survey’s findings wherever possible, and they want to ensure that their study will stand up under fire in cross-examination. In our recent study conducted among lawyers who use research in the courtroom, we heard the following comments and attitudes:

"We attack the methodology . . . try to break the study down. We typically ask for copies of every survey filled out and re-code it ourselves to see whether or not our figures concur with the expert’s report."

"The individual questionnaires are the Achilles’ Heel of the expert. That’s where any weakness in the survey design or its execution can be uncovered and attacked."

"We attack our own research . . . look at each questionnaire and try to find any problems ourselves."

These hostile attitudes, coming not just from your opponent’s counsel but from your own side as well, reinforce the need for precision research. Such an environment compels the expert witness designing the survey and the research firm executing it to be extraordinarily attentive to detail in order to survive unscathed.

Under the legal microscope

The same study identified the parts of the research process that are particularly vulnerable in the courtroom: the questionnaire, the data collection process and coding procedures.

In the questionnaire designed by the expert witness, proper wording of questions is key. In particular, the lawyers who participated in the study identified "leading" and "vague" questions as the most frequent flaws they focus on to attack a survey’s validity. "Leading" questions designed to elicit a specific response and "vague" questions whose responses can be easily misinterpreted will be attacked in the courtroom, accused of producing biased results, and used to punch holes in the survey’s validity.

It is the expert witness’s responsibility to write clear, simple and unambiguous questions and the task of the expert executing the project at the research company to field the questions appropriately. Sometimes a pilot survey is conducted to help identify any problems with wording ahead of time and prevent any issues from coming up in court.

Attention to detail is critical

Even with the perfectly designed survey, problems can arise. During a trial, every detail in the data collection process will be subjected to scrutiny. All types of problems must be looked for ahead of time and eliminated during fielding of the study. The expert executing the research project must make sure no problems arise during the fielding and data processing steps that could be used to undermine the survey’s findings.

For example, one of the lawyers in our study made the following comment about his opponents in a trademark case:

"We were able to show, through the actual questionnaires filled out, that there was a discrepancy in one survey center. The interviewers were supposed to take down verbatims, but it strained credulity that five people in a row would say the same thing."

Because the lawyer was able to show that the data from that one center was suspect, the survey’s results were discredited.

Breakdowns in double-blinding procedures also undermine the results of legal research. If an interviewer or respondent can guess the sponsor of a study, their answers may be biased, attacked in court, and can result in your case being discredited. For example, in one study mentioned by the lawyers, all materials were appropriately labeled with a project code name, but the code included the client’s corporate name – discrediting the study’s results.

Also, a consumer’s shopping experience needs to be accurately duplicated if you are using visual displays to present products for evaluation in a study. For example, for two products to be compared in a trademark infringement dispute, the two packages should not appear on the same shelf in a display during a study, if in fact they appear in different aisles of the supermarket. The actual consumer experience needs to be recreated as closely as possible in order for the survey’s results to be accepted.

Any of these data collection problems, such as interviewer fraud, breakdowns in double-blinding procedures, and inaccurate displays, may be discovered and revealed in court by opposing counsel. The lawyers will quickly point out how this introduces bias to the surveys results, and will use their discoveries to undermine the expert witness’s testimony.

Proper coding is key

Another area of vulnerability in the data collection process is coding. Whenever open-ended questions are used in a survey, the opposition will usually re-examine and attack the coded results. Typically, opposing lawyers will ask for copies of every questionnaire filled out and re-code the answers themselves.

If a respondent’s verbatims are ambiguous or unclear, the opposing lawyers may interpret the responses differently. For example, in a claims substantiation dispute regarding two beverages, one coder may consider "sweet" and "sugary" to be the same, while another coder may see these as different answers.

Such differences in coding should be resolved before the survey is presented. Otherwise coding differences may be brought up during the trial, explored on the witness stand, and used to undermine the credibility of the findings.

Several steps can be taken to avoid coding problems. First, the expert witness can design the survey to only ask open-ended questions when necessary. Second, these open-ended questions can be worded carefully to focus on a very specific point. Third, the research firm executing the study can have two people code the responses to each question and verify each other’s work to help eliminate different interpretations before the results are presented on the witness stand.

A checklist for executing legal research

To avoid these and other data collection problems, our firm uses a special checklist for fielding and processing legal research studies. The purpose of the checklist is to eliminate the element of human error. Some of the rules we follow are:

  • To help reduce the potential of fraudulent responses occurring, no interviewer completes more than one-third of the interviews in a given market.
  • All interviewers conduct a "practice" interview which is documented, signed and returned to us, to confirm that every interviewer is well trained and understands the survey procedures.

Other more obvious rules include: using only pens on questionnaires to prevent erasures which may imply changed answers; creating truly neutral project code names; and removing all references from all materials that may identify the client sponsoring the study to prevent any breakdown in double-blinding. Special quality control procedures are implemented as well, including the minimum validation requirement of 100 percent attempted and 75 percent respondents reached, and 100 percent keypunch verification.

For the expert executing the project at the research company, the checklist of rules, regulations and procedures can go on and on. The overall imperative is to have no errors in data collection or processing. Any problem can be brought up in cross-examination and used to challenge the survey’s findings. The testifying expert witness will have to defend any survey irregularities on the witness stand.

Given the hostile environment and rigorous conditions of conducting research for litigation, it’s not surprising that research firms consider this type of quantitative research to be a specialized niche. However, for market researchers who do conduct legal research, it can be very gratifying to meet the exacting demands of expert witnesses and legal minds and help resolve immediate and important legal disputes in the business marketplace.