Faster than a speeding survey: Part II: The physician's perspective

By Terri Maciolek, Jeffrey Palish | July 1, 2009

Reading time: 5 minutes

Abstract

In the second part of a two-part series on online surveys with physicians, the authors explore doctors’ reasons for participating in the research process and examine the factors that can lead to speeding and cheating.

Research Topics:: Health Care (Healthcare) Research | Incentive Payment & Processing | Medical Interviewing | Online Research | Online Survey Design/Analysis | Online Surveys | Psychological/Emotion Research | Qualitative Research | Quantitative Research | Questionnaire Analysis | Respondent Cooperation/Satisfaction | Survey Design
Industry/Market Focus:: Health Care (Healthcare) | Hospitals | Medical | Physicians
Content Type: Research Methodology

Share Print

Editor’s note: Terri Maciolek is principal and founding partner of Data Quest Analytics LLC, a Wynnewood, Pa., research firm. Jeffrey Palish is regional vice president of Epocrates Inc., a San Mateo, Calif., software firm. This is the second part of a two-part series. The first article appeared in the June issue. The authors thank Critical Mix Inc. for its assistance in data collection, programming and fielding; and Kathleen Tencer, West Chester University intern to Data Quest Analytics.

Is it the driver or the car or both? In part one of this two-part series (“Faster than a speeding survey,” Quirk’s, June 2009), we provided a framework for market researchers to consider protecting online survey data integrity from “drivers” (i.e., respondents) who speed and take shortcuts. We suggested fundamental practices to minimize inclusion of questionable data: 1) validating a respondent’s “license to drive;” and 2) implementing “road traps” (i.e., knowledge traps, speed traps, logic traps, attention traps) to catch those who appear to speed or take illegal shortcuts.

In this article, we delve into the drivers’ (physicians’) perspectives on online surveying. Why do some doctors have a lead foot? Do physician respondents need a refresher course on how to drive? Are the vehicles (survey instruments) to blame for the problems? What can we, as an industry, do to minimize reckless driving?

Good news

What is the driver’s perspective? We queried over 400 physicians using depth interviews and online surveys. The good news from physicians (anecdotally confirmed with a spectrum of survey agencies) is that the incidence of known and intended speeding and cheating by validated physicians is fairly small (with guesstimates around 2-3 percent). Assuming that surveys sample from a reliable panel of verified physicians, it appears that the vast majority of physicians who start a survey do so with the intent to follow the rules of the road and to earn some gas money along the way. Even though financial incentives are key for survey participation among physicians, many also view online surveys as an opportunity to learn about new drugs and treatments as well as benchmark themselves against other clinicians (Figure 1). Indeed, when provided with a well-tuned car (survey) to drive, physicians actually enjoy completing online surveys that ask for their professional opinions and clinical behaviors and perspectives.

While good intentions abound to be safe and careful drivers, all physicians we queried admit that they are not always fully attentive when answering every online survey question. What happens during a survey that causes physicians to be less attentive? It appears that certain parameters of the survey itself affect a respondent’s mindset before the actual survey begins. Survey length is a good example. Physicians indicate that when a survey takes longer to complete than promised, their attention span diminishes (and frustration levels rise). In fact, most indicate that they “sometimes” or “often” become inattentive or less thoughtful when completing an online survey when it takes longer to complete than promised. The result is diminished validity and reliability of the data.

How long is too long from a physician’s perspective? Physicians suggest that about 30 minutes is reasonably acceptable (and shorter is even better), but surveys longer than 40 minutes not only jeopardize participation but attentiveness as well (see Figure 2). Physicians subjectively suggest that fatigue and inattentiveness set in around the 30-minute mark (and sometimes sooner).

Assume responsibility

But what else drives physicians to disengage and/or speed when completing an online survey? The bottom line is that we, as the car manufacturers, must assume responsibility to provide physicians with a vehicle that is well-tuned to perform at the level we expect and require.

In other words, survey design is a critical factor. All physicians we interviewed during our investigation were quite clear that the quality of the survey drives generation of quality data. Like many of us, physicians are busy and constantly multitasking. Still, they take time to complete our online surveys and to carve out quiet time to do so (nearly 70 percent of physicians typically complete online surveys at home, where they contend they can concentrate and give the survey their complete attention). As physicians we interviewed spontaneously and collectively commented, “Physicians are great and deep thinkers . . . we’re highly educated and intelligent . . . a survey that generates valid and reliable data respects that.”

As conscientious market researchers, what must we strive for in constructing and fielding our online surveys to optimize the quality of data generated? Physicians suggest the following (see Figure 3):

• abide by the participation agreement (survey length, topic);

• ask questions and present stimuli that are clear, concise and relevant;

• construct a survey that has a clear and well-organized structure and flow;

• use a variety of questioning formats (multiple-choice, rating, ranking, open-ends, etc.);

• minimize technical issues (frozen screens, visuals that take “too long” to load);

• keep the survey visually pleasing.

More than two in five physicians surveyed indicate that they are “never” or only “rarely” inattentive in answering online survey questions because they are distracted or busy with other matters while trying to complete the questionnaire. Physicians consider the survey experience to be a partnership that should produce a win-win for both respondents and market researchers. In fact, more than 70 percent of physicians surveyed said that they would like to be able to comment (input free text) at the end of a survey as to the quality of the survey itself (what they liked or disliked about it, suggestions for improvement, etc.). Here are some comments physicians shared during the research we conducted for this article:

“The quality of the surveys varies greatly and feedback on this is an excellent way to improve the quality of the survey. Quality in equals quality out.”

“The more complex the survey, the less real data are obtained. It is like Windows Vista vs. Mac. Make surveys fun, entertaining and interactive.”

“I personally tend to spend more time on surveys that are interesting and not lengthy, which allows me to think more and give better/more thought-through answers.”

“I think a more effective way to get good answers to a survey is to make sure it is written well. Time is valuable and if you want doctors to spend the time then make it easy to answer questions accurately and quickly.”

Generate quality data

In our two articles, we’ve explored dimensions of generating and maintaining data integrity in physician-answered online surveys. Much of what we have imparted is of the best practices ilk rather than a discussion of new tools, directions or strategies. High-quality and well-executed survey instruments generate quality data, and checks and balances along the way preserve data integrity. To quote Henry Ford, “A market is never saturated with a good product, but it is very quickly saturated with a bad one.”