Researchers continue making their feelings known in part II of a report on the 2007 Confirmit Market Research Software Survey

'Do something about PowerPoint!'

Editor’s note: Tim Macer is managing director and Sheila Wilson is an associate at meaning ltd., the U.K.-based research software consultancy which carried out the research on which this article is based on behalf of Confirmit. The authors wish to express their gratitude to Confirmit for its permission to publish these results in this article. Part I appeared in the February issue.  

In last month’s issue, we looked at findings from the 2007 Confirmit Market Research Software Survey, analyzing respondents’ answers to questions on industry challenges, the use of in-house or packaged software and plans, if any, to change software in the coming year and the reasons for that change.

This month we explore findings related to mixed-mode research, sample source utilization trends and a wish list of features respondents would love to see in their software.

(The survey, now in its fourth year, was sent to researchers who are actively involved in or responsible for IT within their companies; 70 percent are IT managers, directors or business owners. The 2007 sample represents 233 companies, balanced by region, company size and level of responsibility.)

Mixed-mode

We asked the companies conducting mixed-mode research in 2006 and again in 2007 whether they were able to use the same software platform for the different modes, or were they having to split projects between different software platforms for each mode (Figure 1). Overall, the majority still has to hop between platforms, with all the duplicated effort and hardship that this entails - but this gap is closing. Significantly fewer firms had to contend with split platforms in 2007, and in Europe at least, integrated platforms have now reached parity. America lags behind slightly at 45 percent. The base in Asia-Pacific was too slender for a reliable measure.

We anticipate this shift will continue in the future, even if multimode research remains a minority activity, simply because it is an area that software manufacturers are increasingly supporting in their updated software.

Within the space of a year, there have been developments in requirements for multimode research, as shown clearly in Figure 2. Even though volumes for mixed-mode have not changed, there is a definite shift in demand for the level of support required for hybrid research such as combined CATI and Web research.

We split multimode into three levels of complexity: common authoring (using the same system to design all survey instruments); mixed-mode in parallel (interviewing simultaneously in different modes); and multimode with switching (start interviewing in one mode and finish in another mode).

The demand for the slightly more complex mixed-mode in parallel capabilities has increased at the expense of the more rudimentary common authoring. But demand for the most sophisticated multimode with switching has remained static over the last three years - perhaps not surprising given its daunting technical complexity and the sparse number of software products that can handle this properly.

In North America, the use of multimode with switching stands at 15 percent, which is low compared with Europe (29 percent) and Asia-Pacific (33 percent). As noted earlier, Web-only interviewing predominates in North America , while local issues seem to be preventing its dominance elsewhere. A mixed-mode approach can overcome these issues and this is most likely driving the interest in these parts of the world.

Improving research quality, rather than cost-saving or operational issues, appears to be the primary motivation for research companies conducting mixed- or multimode research (Figure 3). Respondents were allowed to prioritize up to three reasons that came from some earlier research we had carried out. The two that came out on top were to improve sample coverage or representation and to improve response rate. Another similar reason, to be more respondent-friendly, was in fourth place, just behind “client requires mixed-mode” (or “it’s not our doing”!).

The top two reasons between them mopped up most of the votes, with two-thirds of respondents selecting “improve response rates” as one of their three reasons, and over half chose “improve sample coverage.” They were also chosen by over a quarter - 27 percent and 28 percent respectively selecting them as their main reason.

Even though multimode research offers impressive opportunities for cost and efficiency gains, only 8 percent of respondents chose “reduce fieldwork costs” as their top reason; “shorten fieldwork times” and “increase capacity of fieldwork” merited a handful of mentions as the primary reasons. However, “reduce fieldwork costs” does shine through in the total picture, with a respectable showing of 39 percent when second and third choices were added in.

We do wonder whether the general concern over falling response and sampling ills measured earlier in the survey will drive the growth of multimode research, since researchers are clearly aware that mixing modes is a technological trick that can address sample quality and coverage issues and boost response rates in the bargain. The medicine is becoming available: we wait to see if research companies, and their clients, are willing to take it.

Source of sample

Clients still appear to be the most important source of sample for surveys, which perhaps also indicates the extent to which customer research is now underpinning survey research activity, particularly online research (Figure 4). This is despite predictions from the research companies surveyed by us in previous years that client-supplied samples would diminish in importance. However, we may at last be on the point of witnessing this change. If current trends continue, it looks as if access panels may overtake client-supplied samples as the predominant sample source during 2008. Their use has grown steadily in recent years, rising from 54 percent of projects in 2004 to 74 percent in 2007, just three points behind client-provided samples, which, after a long wait, do now appear to be on the wane.

Own panels and specialist sample providers also appear to be making inroads as significant sample sources in 2007, after several near-constant years, although it is too early to be sure whether this is the start of an upward trend.

Distribution modes

Some dramatic changes seem to have taken place in the way that results are distributed since we last conducted this study in 2006. While PowerPoint is still, by far, the most popular delivery method (well over one-half of all projects appeared in PowerPoint in 2006), by 2007 this has slipped to 48 percent. The use of paper or near-paper formats - Word documents, printed tables and Acrobat PDFs - has also nose-dived (Figure 5).

Only online results delivery has held its own, and in the case of interactive analysis, has edged up from 8 percent in 2006 to 10 percent in 2007. It is also clear that market researchers are distributing results in multiple formats. But perhaps as clients and market researchers are becoming more savvy with online methods, it looks as if some of the more traditional methods are on the wane. Certainly, the proportion of studies being distributed in multiple formats appears to have declined during 2007.

Despite this, modern desktop analysis tools have yet to stifle demand for printed crosstab reports. Although around one in seven of companies (14 percent) are using bulk crosstab reports, we learned from another question we asked that 52 percent of research companies still consider it essential to be able to produce them. Nevertheless, the trend seems to be downward. The same question in 2004 revealed that 59 percent of research companies considered bulk crosstabs to be a vital deliverable.

Analysis and reporting - future wishes

Market researchers seem to be giving software developers two very clear messages in a question that asked about their top three wishes for analysis and reporting tools (Figure 6).

Firstly, “Do something about PowerPoint!” Nearly three quarters of respondents rated this as one of their top three priorities. While many packages will create graphics that are ready for PowerPoint, or will transfer data directly into PowerPoint, only a handful actually automate the process so as to eliminate the painstaking manual finessing that is necessary to turn raw charts into presentable output.

Secondly, our respondents give online analysis tools the thumbs down, with over three out of five of them seeking better online analysis tools. From our own knowledge of the tools on offer, we know this area to be a mixed bag. Some of the analytical tools provided on the back end of data collection products lack luster and are either limited in their capabilities or are horribly complicated to use. There are, however, a few exceptions in the integrated tools plus a growing number of dedicated online analytical platforms that happily receive data from numerous sources. It seems there are opportunities for sellers of Web-based crosstab tools in 2008. There is a similar, though less emphatic opportunity in the 14 percent making a plea for better desktop analytical tools.

We can probably consider the other products or features on the industry’s shopping list as “nice to have” - certainly worthy of consideration by the industry’s software providers as they spruce up their online and desktop analytical products.

Continuous research

The 2007 survey also revealed the importance of continuous research projects to the industry, with companies around the world attributing 23 percent of their activities to work on trackers: 26 percent in North America, 25 in Asia-Pacific and 21 in Europe . As Figure 7 shows, it is an area of activity that brings its own special set of problems, not the least of which is the managing of all the inevitable changes that occur.

However, as few of the problems cited in this question are managed in mainstream MR survey software, we were very surprised to observe that 70 percent of research companies worldwide were either highly or moderately satisfied with how their software handled continuous research. This is all the more surprising as only 4 percent of companies claimed they had fully automated the management of their continuous projects, 32 percent said they had achieved partial automation (still with some manual intervention) and 63 percent confessed to little or no automation in the handling of continuous projects. This has not only cost but quality repercussions which research buyers would be wise to pursue with their data suppliers when their continuous contracts next come up for renewal.