Online research still reigns supreme

Editor’s note: Tim Macer is managing director, and Sheila Wilson is an associate, at meaning ltd., the U.K.-based research software consultancy which carried out the study on which this article is based on behalf of Globalpark.

The findings from the 2009 Globalpark Annual Market Research Software Survey offer some unique insights into how the research industry goes about its business; how it is looking to the future and some of the concerns it has today. It also puts into perspective some of the latest trends in the industry. This year’s study took a look at mobile research and communities.

Largely it finds the jury is still out on mobile research, and some way from reaching any consensus on its viability, with proponents and detractors in equal measures. We also learned that, despite the attention that research communities are commanding, actual working research communities are still surprisingly thin on the ground.

The survey is now in its sixth year and that means some of the survey’s tracker questions are revealing some interesting long-term trends. As we conducted this research in a recession, we looked to see if it was having an impact on technology and processes. We found little in the way of direct evidence in attitudes or approaches. Indirectly, though, it perhaps manifested itself in a much greater reluctance to participate in the survey, despite us casting our sampling net much wider, and involving two panel companies to supplement our own sample. We failed even to make our target of 200 completes in the survey, achieving only 188 this time.

It’s a 15-minute online survey among senior decision makers involved in technology in research companies in three global regions: North America, Europe and Asia-Pacific. We allow only one participant from any company, or, in the case of multinational companies, in that country or region. We don’t claim the survey is statistically representative, as ours is clearly a convenience sample, but we do still feel it offers some useful insights into what is actually happening in the industry away from the spotlights of publicity.

We’re very grateful to Globalpark, who sponsored the 2009 survey in an exemplary manner, and also to the generosity of our participants for letting us share their thoughts with you. For this article, we have picked eight topics we thought might be of interest. There’s more in the full report, though, and it’s available free of charge at www.meaning.uk.com/gmrss2009.

Modes of research

Each year, we ask respondents which modes of research their companies offer (Figure 1). Web reigns truly supreme, with nearly all respondents (94 percent) in 2009 saying that they provide online surveys. However, for the first time, the proportion of companies offering Web research has leveled off.

Although the percentage of firms offering CATI has remained at around 70 percent since 2006, volumes for CATI appear to be declining. In 2006 CATI provided 27 percent of total quantitative revenues and by 2009 this was 23 percent. Over the same period, Web has grown from 40 percent to 46 percent in our tracker. We need to point out that this is higher than in some other industry studies such as the ESOMAR Global Market Research report. In a further question, we asked respondents about their growth expectations for the future. Each year since 2006, companies have been predicting a small decline, and that predicted decline is growing larger each year.

With over one-third (37 percent) of market research companies offering mixed-mode CATI and Web, it can be classed as a mainstream methodology but it does not appear to be developing. This seems slightly surprising given its obvious potential for increasing response rates and is probably a function of its greater complexity and cost, though methodological concerns remain an issue for many researchers.

The minority methodologies, especially SMS and “other mixed-mode” are growing, albeit slowly. Although not shown in Figure 1, our results indicate that large companies and those in North America are at the vanguard with these more unusual techniques, so perhaps these companies are pursuing a diversification strategy - nearly one in five (19 percent) of large companies offer SMS and over a quarter (27 percent) offer IVR.

These less-commonly practiced methodologies may offer lots of promise, but it has to be remembered that most (86 percent) of the revenues from quantitative research are from just three methodologies: Web, CATI and paper. The same has been true since we first started measuring this in 2006.

Mobile interviewing

Every year we introduce topical questions that reflect the buzz in the industry. In 2009, we focused on mobile interviewing - mobile self-completion on a smartphone, iPhone, BlackBerry and the like, as opposed to mobile CAPI research. In this question we asked what kinds of mobile research offer the greatest potential for growth (Figure 2).

Our respondents thought that all technologies had potential, but rated HTML Web surveys delivered via the mobile device’s Web browser above the others. However, large companies slightly favor surveys via applets. It’s a more high-tech approach that gives a richer and more controlled interviewing environment, but it requires a greater level of commitment from the participant, who needs to agree to download the applet to their device.

In fact, large companies are more optimistic about all methodologies, which probably tells software vendors where to target their sales efforts! And this appears to fit in with results elsewhere in the survey, which show that large companies appear to be far more positive about the viability of mobile research as a whole.

There are clearly advantages to mobile research, but, in another one of our mobile research questions, we saw that the respondents view the main advantage of it being an additional way of reaching respondents and filling up quotas, although “faster turnaround” is seen as important by over a quarter (28 percent) of those who took part in this study.

Major challenges

When asked about the major technical or operational challenges in mobile research (Figure 3), almost two-thirds of respondents felt that the biggest challenge is the way the small format of a mobile device limits the type of questions you can ask. Yet the arrival of the iPhone raised the game and showed that many of those constraints fall away if the questionnaire is designed for that format and not simply ported over from a standard Web survey. (If only everyone had an iPhone!)

Presumably, over the years, others will catch up and more applets will appear, thus offering some relief over current limitations, although it is clear that researchers are always going to have to address the issue of less space with more creative and less-wordy questionnaire designs.

The ever-present difficulty of obtaining sample is the second-most important challenge, with over one-half of respondents highlighting this. But then, that was often cited as an insurmountable problem for online research in its early days.

Support for different devices and complex cost structures of different networks is a minefield, especially when conducting international research. We are quite surprised these challenges are not even higher up the list.

“Not well-supported in the software” is relatively low down the list of challenges but we suspect that this lies at the second hurdle, once the viability of the channel is accepted. It should not be read as a vote of confidence in the software. The actual tools on offer are largely from niche players, and there is a limited choice available. Only a handful of the online or mixed-mode data collection suites offer integrated support for mobile at present. None of this is surprising given that the slice of the action is less than 1 percent of quantitative revenue, but demand for these applications is bound to increase, we predict.

Online communities

The survey also looked at how many market research companies in our study are operating online communities (Figure 4). Online communities have clearly not yet taken off in a way you might anticipate from the buzz that they are creating at conferences and in the research press, with only one in six companies (17 percent) operating any online communities. Indeed, over one-half (56 percent) have no plans to start any online communities.

Even among the minority who are running online communities, they are still in the early-adoption stage. We learned in a follow-on question that 59 percent of companies are operating three or fewer communities and 35 percent are operating only one. Speaking to practitioners, it is clear that a major issue is the time and effort it takes to run even one community - and when budgets are as tight as they are today, it’s hard to sell this work and make a profit.

Large companies and those in Asia-Pacific seem further ahead, since these two groups have a noticeably higher proportion of companies who already operate communities or have them in development. In contrast, communities seem to be very rare in small companies and in Europe, with only 10 percent and 11 percent, respectively, of these companies already running communities.

Our general impression is that most of the industry is holding back from online communities, even though there is a reasonable amount of technology available now that is specifically designed for market research communities.

To try to tease out whether the industry does differentiate between panels and communities, we asked respondents how they felt communities differed from conventional panels. The top three differentiators cited were that members interact with each other (51 percent), that they focus in an area of particular interest to the respondent (49 percent) and that respondents are more motivated to participate (43 percent). Rather tellingly, over a third (38 percent) of respondents said that community research is not as rigorous as panel research. Assuming that the make-up of our sample is overtly aligned with quantitative research, and that the very next ranked item was that communities are “more qualitative than quantitative” we read this as the oft-rehearsed accusations about rigor that fly from one camp to the other. Neither approach is immune to poor research design, and we can see no reason why communities cannot be administered rigorously. The dangers come in claims over representativity. But that should not devalue the insights that communities can provide, if research buyers can be persuaded to pay for them.

Changing software

One of our annual tracker questions asks respondents whether they are likely to change their software in the next year or two (Figure 5). Interestingly, even in a recession, a lot of companies were still planning to make a switch, with almost one-third (32 percent) wishing to change their software and a further quarter (26 percent) reviewing the situation (expressed as “undecided”). Only 43 percent plan to keep their existing software.

Every year we see that large companies are more likely to say that they plan to change their software, with over two-fifths (43 percent) in 2009 saying yes, compared with 31 percent of small companies and 21 percent of medium-sized companies. Small and medium-sized companies were less likely to change their software - 45 percent of small and 49 percent of medium-sized companies state no plans to change.

As we have seen in many questions in this iteration of the survey, the large companies are more pro-change. Large market research businesses are getting larger and this growth tends to necessitate changes in technology. Presumably, large companies are also in a better position to experiment with new technologies because they are more likely to have specialist staff and they certainly have more financial clout. But this flies in the face of an assumption many may hold that the smaller companies, being more nimble and less bureaucratic, are therefore more innovative. On the contrary, it seems that the majority are more risk-averse.

What type of platform?

Another of our annual tracker questions is the extent to which research firms use an integrated software platform for online and other interviewing methods or use different platforms (Figure 6). There is a clear and sustained trend over the last four years that shows a shift toward using an integrated platform for multiple methodologies, and that seems set to run on for a little while yet.

We also asked respondents how important mixed-mode support was to them if they were buying new software. As in 2008, when this question was first introduced, nearly all respondents (84 percent) thought that it was important (moderately important or above) when choosing a new data collection tool. This is clearly food for thought for software developers!

Sample sources

We asked respondents to tell us what proportion of their Web survey research relied on different sample sources, using revenue as our metric. As shown in Figure 7, the results vary by company size and region. Altogether, panels account for at least 61 percent of work by value. Client samples account for one quarter (24 percent).

Large firms appear to have invested much more in developing their own panels, judging by the much larger volumes they have shifted to their own proprietary panels: 46 percent, which is close to half their volumes, against 17 percent for smaller firms, for whom such investment is obviously more difficult. The overall pattern is little changed from the figures we obtained in 2008. Looking at this, it is perhaps a little surprising that large companies do not run more online communities (see Figure 4).

As in 2008, projects with sample from specialist providers contribute a small proportion of volume (13 percent). This is low considering that, in another question in which we asked the companies to state which sources of online sample they use, almost one-half of firms report using specialist sample providers. A similar number of companies use their own panels, yet the contribution from own panels is much higher, at 26 percent. Perhaps companies are favoring their own panels due to the greater control and greater potential for profitability.

Interestingly, in the question about the number of companies who use each sample source, it can be seen that all sample sources, except for own panels, are used more in North America than in other regions - and this is especially the case with access panels. Perhaps this is indicative of lower completion rates in that region.

Gauging future demand

To gain insight into the future demand for some of the online analysis and reporting systems, we asked respondents to say to what extent they anticipated an increase in demand over the next year.

There appears to be plenty of demand for online reporting of all types. Virtually nobody surveyed was anticipating any decline. As is the case with many questions in this study, large companies are more advanced at anticipating change. At the detailed level (which we have not shown in Figure 8), large businesses anticipate a greater increase in demand than small and medium-sized companies in every area of online reporting. For example, 97 percent of large companies predict an increase in demand for external portals, against 55 percent of small companies.

Consistent with this, the large companies are expecting the greatest increase in demand in the more high-tech areas, such as portals, and the smallest increase in demand in the least technically-demanding delivery methods, such as fixed online reports.

This is clearly another area where software developers need to take note. The demand in the market research industry is there, but the tools available still don’t make it easy to publish data in a dynamic and Web-enabled way. Demand appears high for digital dashboards, for example, yet very few of the mainstream packages provide much practical support for creating them. If custom solutions are required - an increasingly common request in project briefs - this ultimately pushes up the cost to the client and diminishes profit to the research provider.