A report on the 2006 Confirmit Market Research Software Survey

Online makes more inroads

Editor’s note: Tim Macer and Sheila Wilson work for London-based consulting firm meaning ltd.

Now in its third year, and reported in Quirk’s for the second time, the 2006 Confirmit Annual Market Research Software Survey provides a unique snapshot of the trends and changes happening in the marketing research industry’s use of technology. The survey is carried out independently and anonymously on behalf of Oslo-based research software maker Future Infor mation Research Management (FIRM) by London-based consulting company meaning ltd. following professional research standards. The 2006 results are based on a representative sample of 213 senior managers or owners of research companies around the world, with a large representation from North America.

Researchers may not yet rank with physicians on the hard-to-get scale, but they still proved to be relatively elusive as survey respondents. However, we were reasonably happy with the 24.5 percent response rate we received, which proved to be well-balanced by area of responsibility, company size and global region.

We are most grateful to FIRM, who commissioned this research, for their permission to share these results with the readers of Quirk’s. A longer report on the survey is available for download free of charge at www.confirmit.com/download/documents/research.

Online is ubiquitous

The results of the 2006 survey show that online research and the software to do it is now virtually ubiquitous even among the smallest research companies. They also chart the recent growth of mixed-mode research and indicate that that growth will likely continue strongly.

In our report, we will reveal some surprises in the reasons why companies are using mixed methodologies. We also expose some interesting global differences in how companies source their samples online and examine the rapid growth of emerging panel management software.

For the software suppliers, there is either encouraging or chilling news that a sizeable chunk of research companies are considering dumping their incumbent software for something offering greater flexibility and improved efficiency, amongst other reasons. As turmoil sets in for the suppliers, a crumb of comfort is that cost appears not to be the driver.

• Research modes cited

In 2006, more research companies than in 2005 said they are using CATI and CAPI. In 2005, 65 percent worldwide used CATI and in 2006 this has risen to 70 percent. The figures for CAPI are 25 percent in 2005 and 30 percent in 2006. Admittedly, these are not major increases, but still they represent some growth where none was anticipated. It will be interesting to see if the numbers continue to rise when we repeat the survey in 2007.

There are no signs of paper surveys disappearing: two-thirds of companies still relying on them. But the emerging area is mixed-mode, which we asked about for the first time this year. At present it is probably best described as being important to a very sizeable minority of companies worldwide (41 percent). We found the most popular combination is, as suspected, Web and CATI (36 percent) with only 15 percent are doing other combinations of mixed-mode research such as Web/paper, CATI/CAPI and so on.

The extent to which either CATI or CAPI is offered closely follows company size. While many in North America have written off CAPI as a method, it is still practiced by 54 percent of large companies, but only 23 percent of small companies. However, even this figure is surprising, as our definition of small companies is those with revenues up to $1.5 million.

SMS research - using instant messaging on cellular phones - does not seem to have taken off, with a reduction in the proportion of companies offering this service - 6 percent in 2005, down to 3 percent in 2006. But it is worth noting that surveys on paper are still conducted by almost two-thirds of those in this study, with almost no variation by the size of the company.

• Research mode by volume

After asking which modes they had the facility to undertake, we also asked for the volume by each mode, using revenues as the measure rather than numbers of interviews. Figure 2 clearly shows where research companies earn most of their money: Web, CATI and paper together are the source of 88 percent of revenues from quantitative work.

It is in volume that CAPI appears truly diminutive, at 5 percent of volume worldwide, with mixed-mode surveys accounting for a slightly greater share (7 percent). However, with only one in three companies offering CAPI, against two in five offering mixed-mode, it looks as if there is more CAPI work around for the current capacity than there is mixed-mode work, at present. But for a method that many consider to be experimental still, 7 percent is a decent showing. It appears that this is an area where companies are offering the service now in anticipation of future returns. With CAPI, the situation is less clear. It accounts for as much as 12 percent of the revenues of larger research firms but only 4 percent of the smallest ones. Has CAPI research quite simply reached a plateau? If that is the case, we are curious as to why more companies seem to be offering it than in past years.

Several large research companies have made it clear in various public announcements they wish to dump the “old” methods in favor of lower-cost and faster online surveys. How interesting it is to find that it is the medium-sized companies that seem to have achieved this balance. A whopping 55 percent of their volume is online, with everything else in the teens or single digits. Among large firms it is a fairly equal three-way split between CATI, Web and “the rest,” with CATI very slightly in the lead still.

Among the smaller and independent operators, Web is also taking the lead, nine percentage points ahead of CATI and paper. Perhaps an example of the advantages of piloting a sailboat rather than an oil tanker.

• Packaged software in use

Figure 3 should have developers of market research software scratching their heads in bemusement. Although most market research firms use only packaged software bought from a supplier, one in five of the developers’ potential customers are using software they have written themselves for online or CAPI work. For CATI, one in every 10 system out there has been knitted at home.

We continue to find it astonishing that so many market research companies are straying so far from their core strengths to develop and maintain their own software when there is such an abundance of applications of every kind on the market. The only justification can be where the demands are so unique that existing solutions cannot meet them. But we wonder just how unique can any one company’s data collection methodologies be in a world where the research buyers are demanding more consistency across the research they buy.

This is a market where many products in daily use are very long in the tooth. A lot of pre-Windows tools are still being used, for example. Compared to these tools, it is not difficult to build something in-house which is better. But that ignores the many products now available that use complex but ultimately powerful tiered architectures. While the result for the end user may be more versatile and easier-to-use software, the task for the in-house developer, without the resources of a software company behind him or her, gets harder.

• Changing software in the next one to two years

In 2006 we asked respondents for the first time whether they had plans to dump their existing software anytime soon. The strength of the “yes” we got took us aback. In Figure 4 you can see that only half (49 percent) of all companies are planning to keep their current software over the next one to two years. A quarter are planning to change their software and another quarter are undecided. Within large companies, these figures are even more striking, with under half (44 percent) planning to keep their software, but nearly two in every five are planning to buy new software within the next one to two years.

This shows that software manufacturers cannot take their customer base for granted because within a short time, many of their customers, especially the large ones who generate the most revenue, intend to be using someone else’s software.

This could again be influenced by the slow pace of change in some of the tools widely used today, which have not kept pace with modern developments, and are showing their age. Fortunately, our next question was “why?”

• Reasons for considering a change in software

Our list of reasons (Figure 5) was derived from some qualitative research we had done earlier. The reasons cited are diverse and well-spread, though the leader is unsurprising. We considered it much more interesting that cost comes well down the list. This is at odds with the experience of most suppliers. Partly this can be attributed to the high cost of moving from one system to another: It is an expensive way to save costs. But Figure 4 shows the backdrop, where despite the cost of making a change, a quarter say they are likely to do so.

So beyond getting a better model and saving some money, the really revealing reasons are that more than half of the potential switchers are doing so to achieve efficiency improvements through automation and to modernize. By implication, they are saddled with tools which are labor-intensive, old-fashioned and not integrated with other tools or interviewing modes.

Many companies also want to use the same tool across all their activities or in all their offices (47 percent cited consolidating on one platform as a reason to change), which could be due to aspirations to having a mixed-mode solution, if not even extending to doing mixed-mode research, or be as a result of company merger and consolidation, of which there has been a lot of late.

At least companies do not feel constrained by company policy, as this was cited by just 16 percent of the sample. Though it came out as a minority reason, one in five companies were not happy with their existing supplier (not the software).

There are a diverse range of reasons for changing software, all of which are important. The most often cited reason for changing software is to seek “more flexibility, more capabilities or better functionality.” Nearly two thirds of respondents chose this.

The reasons cited do seem to suggest that a decent proportion of market researchers place value on innovation and are not afraid of change. The top reasons typify aspirational thinking towards improvement, rather than reactive thinking brought on by being forced to change.

• Reasons for mixed-mode research

This is the first time we have asked respondents why they are doing mixed-mode research. Our results (Figure 6) show that contrary to common opinion, the main reason for mixed-mode research is not to cut fieldwork costs (although this is important to 42 percent of the sample) but the quality-related issues of increased response (both response rate and sample coverage), and improving the respondent experience - two factors which are closely aligned with combating the acknowledged decline in response rates and ensuring that research data are reliable.

Tellingly, 42 percent of the time it is the client who is driving the decision by insisting that the survey uses a mixed methodology approach. This perhaps reveals a somewhat reactive approach in the industry to mixed-mode methodology still.

We were heartened to see that 13 percent of our sample had the courage to admit that a mixed-mode pitch still looks good in proposals, among those early adopters that can do it efficiently without it pushing up the costs.

• Which sources of online sample do you use?

Our next three figures focus on panels: who is using what, how these are changing at present and what kinds of volumes are involved.

Though abundantly mentioned, the fact that agencies are widely using panels from their clients is not interesting in itself until we look at the volumes later on. Access panels continue to be more widely used in North America than they are in other global regions, where own panels are much more prevalent (Figure 7).

Around a third of companies in North America now have their own panels. But they are behind the rest of the world in this respect, with 53 percent and 56 percent in Europe and in Asia-Pacific, by comparison, having their own panels.

• Online sample utilization trends 2004-2006

In our previous two studies, respondents consistently anticipated making less use of client-supplied sample in the future. But this remains an unfulfilled aspiration. The utilization of client sample actually increased in 2006 (it was 75 percent and 73 percent in 2004 and 2005 respectively, and 82 percent in 2006), showing a widening gulf between hopes and actions. Previously, it also seemed that access panels were gaining in popularity and the 2006 results confirm the ascendancy of the rented panel. In 2005, 57 percent used access panels, whereas 70 percent used them in 2006 (Figure 8).

In 2005, our respondents predicted a modest growth in the use of own panels, but this too has not been the case: it was just under 45 percent then and still is now. The use of specialist sample providers has also remained static.

It is not clear to us why this gap between expectations and reality persists. There is likely to be continuing demand for research companies to work with samples supplied from their clients - almost always customer lists, which cannot generally be obtained from any other source. But that does not explain why the dream to build their own panels is not being realized.

• Sample sources by volume

We already know that access panels and samples from clients are very widely used. What surprised us was the large proportion that client-provided samples accounts for: 31 percent globally and 29 percent in North America (Figure 9).

The specialist sample providers, though also widely used, supply only a minority (11 percent) of the total sample volume - a figure that does not seem to vary much by region.

There are interesting regional swings in the use of access panels and own panels. As seen in the utilization levels, access panels are popular in North America and far less so in Europe. The volumes actually amplify this difference, with North America, at 41 percent, driving through twice the volume of their European counterparts, with a rather meager 19 percent.

Turn to own panels and the situation reverses, with only 15 percent of volumes in North America coming from proprietary panels, against 33 percent in Europe and even 27 percent in Asia-Pacific, where the two sources seem to be more in equilibrium.

There is an abundance of global access panels now, so the low uptake in Europe is more likely to be due to active choice or other considerations such as control, quality or cost, rather than scarce availability.

• Software tools used for panel management

Figure 10 appears to show a golden opportunity for software developers, with very few people using off-the-shelf panel management tools. Nearly half of respondents report using their own in-house software for panel management. And a sizeable minority are still using office automation tools such as Microsoft Excel or Access, which offer poor quality control, no audit trail and considerable scope for human error.

There has, until recently, been very little choice in commercially available software tools, and the functionality provided in online survey tools too often lacked luster, and many were more about connecting the panel to the survey rather than actively managing the panel. Hence, many people have built their own tools or made do with Access and the like.

However, as Figure 11 shows, this is now changing rapidly.

• The growth in panel management software

One of our more dramatic findings this year is the marked rise of packaged solutions for panel management, and the extent to which this is displacing the own-grown tools.

The panel management opportunities for software manufacturers were probably better last year, but remain good for a few years to come. For anyone developing a standalone product for panel management, they may not be as golden an opportunity as first appears. Closer examination of Figure 10 shows that there is already a clear move away from using in-house solutions and towards using the panel management capabilities in software provided by software developers. Nevertheless, the use of supplied panel tools has doubled in two years.

• How important are printed crosstabs?

In Figure 12, we look at the analytical side. For those of you dying to stop shifting large piles of paper around your workplaces and save trees, disappointment is clearly at hand. Most respondents still consider printed crosstabs essential or moderately important. This figure has actually slightly increased since 2005.

Though crosstabs are still considered important, it was shown in another question that they are now a minority delivery method because less than one in four (23 percent) of projects continue use crosstabs as a delivery method. PowerPoint slides continue to be the most popular deliverable, accounting for 48 percent worldwide and 52 percent in North America. True electronic methods remain in the minority: static online reports are used on 22 percent of projects in North America and in Europe, but just 11 percent in Asia-Pacific. Interactive delivery via data portals or specialist software accounts for only 11 percent of volumes worldwide, and 10 percent in North America.